[HADOOP-16248] MutableQuantiles leak memory under heavy load

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.9.2
• Fix Version/s: 3.0.4, 3.2.1, 2.9.3, 3.1.3
• Component/s: metrics
• Assignee: Alexis Daboville
• Reporter: Alexis Daboville
• Created: 11/Apr/19 08:41
• Updated: 17/May/19 14:41
• Resolved: 17/May/19 14:29

Description

In some circumstances (high GC, high CPU usage, creating lots of
S3AFileSystem) it is possible for MutableQuantiles::scheduler [1] to fall
behind processing tasks that are submitted to it; because tasks are
submitted on a regular schedule, the unbounded queue backing the
ExecutorService might grow to several gigs [2]. By using
scheduleWithFixedDelay instead, we ensure that under pressure this leak won't
happen. In order to mitigate the growth, a simple fix [3] is proposed, simply replacing scheduler.scheduleAtFixedRate by scheduler.scheduleWithFixedDelay.

[1] it is single threaded and shared across all instances of MutableQuantiles: https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MutableQuantiles.java#L66-L68

[2] see attached mutable-quantiles-leak.png.

[3] mutable-quantiles.patch

Issue Links

• is depended upon by: HADOOP-15620 Über-jira: S3A phase VI: Hadoop 3.3 features
• is related to: HADOOP-16278 With S3A Filesystem, Long Running services End up Doing lot of GC and eventually die