Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-17209

Erasure Coding: Native library memory leak

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.3.0, 3.2.1, 3.1.3
    • Fix Version/s: 3.2.2, 3.3.1, 3.4.0
    • Component/s: native
    • Labels:
      None

      Description

      We use both apache-hadoop-3.1.3 and CDH-6.1.1-1.cdh6.1.1.p0.875250 HDFS in production, and both of them have the memory increasing over -Xmx value. 

       

      We use EC strategy to to save storage costs.

      This's the jvm options:

      -Dproc_datanode -Dhdfs.audit.logger=INFO,RFAAUDIT -Dsecurity.audit.logger=INFO,RFAS -Djava.net.preferIPv4Stack=true -Xms8589934592 -Xmx8589934592 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled -XX:+HeapDumpOnOutOfMemoryError ...

      The max jvm heapsize is 8GB, but we can see the datanode RSS memory is 48g. All the other datanodes in this hdfs cluster has the same issue.

      PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 
      226044 hdfs 20 0 50.6g 48g 4780 S 90.5 77.0 14728:27 /usr/java/jdk1.8.0_162/bin/java -Dproc_datanode

       

      This too much memory used leads to my machine unresponsive(if enable swap), or oom-killer happens.

       

        Attachments

        1. datanode.202137.detail_diff.5.txt
          18 kB
          Sean Chow
        2. HADOOP-17209.001.patch
          2 kB
          Sean Chow
        3. image-2020-08-15-18-26-44-744.png
          14 kB
          Sean Chow
        4. image-2020-08-20-12-35-39-906.png
          29 kB
          Sean Chow

          Activity

            People

            • Assignee:
              seanlook Sean Chow
              Reporter:
              seanlook Sean Chow
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: