Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-16248

MutableQuantiles leak memory under heavy load

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.9.2
    • Fix Version/s: 3.0.4, 3.2.1, 2.9.3, 3.1.3
    • Component/s: metrics
    • Labels:
      None

      Description

      In some circumstances (high GC, high CPU usage, creating lots of
      S3AFileSystem) it is possible for MutableQuantiles::scheduler [1] to fall
      behind processing tasks that are submitted to it; because tasks are
      submitted on a regular schedule, the unbounded queue backing the
      ExecutorService might grow to several gigs [2]. By using
      scheduleWithFixedDelay instead, we ensure that under pressure this leak won't
      happen. In order to mitigate the growth, a simple fix [3] is proposed, simply replacing scheduler.scheduleAtFixedRate by scheduler.scheduleWithFixedDelay.

      [1] it is single threaded and shared across all instances of MutableQuantileshttps://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MutableQuantiles.java#L66-L68

      [2] see attached mutable-quantiles-leak.png.

      [3] mutable-quantiles.patch

        Attachments

        1. mutable-quantiles-leak.png
          279 kB
          Alexis Daboville
        2. mutable-quantiles.patch
          0.8 kB
          Alexis Daboville
        3. HADOOP-16248.00.patch
          0.8 kB
          Alexis Daboville

          Issue Links

            Activity

              People

              • Assignee:
                adaboville Alexis Daboville
                Reporter:
                adaboville Alexis Daboville
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: