Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8896

DataNode object isn't GCed when shutdown, because it has GC root in ShutdownHookManager

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • None
    • 2.8.0, 3.0.0-alpha1
    • test
    • None
    • Reviewed

    Description

      The anonymous Thread object created in ShutdownHookManager is a GC root.
      screenshot_1 shows how DN object be traced to the GC root.

      It's not a problem in production.
      It's a problem in test, especially when MiniDFSCluster starts/shutdowns many DNs, which could cause OutOfMemoryError.
      screenshot_2 shows many DN objects are not GCed when run the test of HDFS-8838.

      Attachments

        1. HDFS-8896.01.patch
          2 kB
          Walter Su
        2. screenshot_2.PNG
          3 kB
          Walter Su
        3. screenshot_1.PNG
          38 kB
          Walter Su

        Issue Links

          is related to

          Sub-task - The sub-task of the issue HDFS-8838 Erasure Coding: Tolerate datanode failures in DFSStripedOutputStream when the data length is small

          • Major - Major loss of function.
          • Resolved
          relates to

          Sub-task - The sub-task of the issue HADOOP-12261 Surefire needs to make sure the JVMs it fires up fit within the memory available

          • Major - Major loss of function.
          • Patch Available

          Activity

            People

              walter.k.su Walter Su
              walter.k.su Walter Su
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: