Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-10840

Fix OutOfMemoryError caused by metrics system in Azure File System

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.4.1
    • Fix Version/s: 2.7.0
    • Component/s: metrics
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      In Hadoop 2.x the Hadoop File System framework changed and no cache is implemented (refer to HADOOP-6356). This means for every WASB access, a new NativeAzureFileSystem is created, along which a Metrics source created and added to MetricsSystemImpl. Over time the sources accumulated, eating memory and causing Java OutOfMemoryError.

      The fix is to utilize the unregisterSource() method added to MetricsSystem in HADOOP-10839.

        Attachments

        1. HADOOP-10840.2.patch
          7 kB
          shanyu zhao
        2. HADOOP-10840.1.patch
          6 kB
          shanyu zhao
        3. HADOOP-10840.patch
          4 kB
          shanyu zhao

          Issue Links

            Activity

              People

              • Assignee:
                shanyu shanyu zhao
                Reporter:
                shanyu shanyu zhao
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: