[SLING-10133] Memory leak in MonitoringDistributionPackageBuilder

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: Content Distribution Core 0.4.0
• Fix Version/s: Content Distribution Core 0.7.0
• Component/s: Content Distribution
• Assignee: Unassigned
• Reporter: Timothee Maret
• Created: 10/Feb/21 12:45
• Updated: 08/Mar/22 12:56

Description

The MonitoringDistributionPackageBuilder maintain a list of MBean for the latest packages. The number of packages to be monitored is passed as the queueCapacity via the constructor. When the queueCapacity is 0, the monitoring is disabled.

VaultDistributionPackageBuilderFactory and DistributionPackageBuilderFactory disable this feature by default. An environment that runs for multiple weeks without restart and with the default configuration will experience a memory leak that leads to the JVM running out of memory.

The implementation has two flaws that explain the memory leak.

 

#1 - Registering a MBean when the queueCapacity is 0

The code unconditionally registers a MBean even if the queueCapacity is 0. We need to only register a MBean when the capacity is > 0.

#2 - Concurrency issue when un-registering MBean

The code attempts to remove by checking if the queueCapacity equals the list of MBeans. This check works in a single threaded context but it falls short when registerDistributionPackageMBean is invoked concurrently. In the latter case, it can happen that the check never holds true leading the mBeans queue to grow indefinitely. One solution is to leverage the features of the LinkedBlockingDeque. Create a LinkedBlockingDeque with bounded capacity and rely on the returned status from the offer method to decide if an item needs to be removed.