After analyzing the code of the JMXMP connector, it turns out that there is, in fact, no leak. The connector uses a notification buffer designed as a circular buffer where notification references are replaced when the buffer is full, but never explicitely released otherwise (apart from when the connector is closed). For instance, if the buffer size is 20, then the first notification will be kept until 20 more notifications are emitted, at which time it will be replaced with the 21st. The main goal of this design is to minimize the risk of losing notifications, at the expense of increased memory usage.

The notification buffer size can be specified as a system property with "-Djmx.remote.x.notification.buffer.size=<size>", which allows a mitigation of the impact on memory usage.

Consequently, I will change this bug report into an enhancement request and explore ways of minimizing the memory usage of the JMX notifications.

Currently, I'd like to explore these possibilities:

• erase notifications as soon as they have been sent to at least one listener. Unfortuately, this doesn't preclude the possibility of OOME, it depends on whether any listener subscribed to the notifications and on the frequency at which notifications are emitted
• use soft references in the notification buffer: this guarantees that all notifications are garbage-collected before an OOME is raised, but increases the risk of missing notifications
• offload the notification's user data to external storage (disk, database, etc.) when memory usage reaches a configurable threshold. This is what we do for jobs received by the JPPF driver. This solution avoids OOMEs and does not increase the risk of missing notifications. Of course, offloaded notifications will have a performance impact since they will be much slower to write/read to/from external storage. No magic here. It also has the advantage that it can be implemented without changing the JMX connector code, since we can "simply" customize the handling of the user data in our notification class (org.jppf.management.TaskExecutionNotification).

Finally I went with a solution which offloads the user data of a notification to disk, when the used heap memory reaches a configurable threshold. In the process, I also noticed that the object wrapping defined for the JMXMP connector was needlessly serializing objects to memory, so I also changed that. It is now possible to emit notifications from the tasks such that the sum of their retained memory is higher than the heap size.

Implemented in:

• trunk revision 4281
• branch b5.2 revision 4282