Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-7470

SecondaryNameNode need twice memory when calling reloadFromImageFile

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      histo information at 2014-12-02 01:19

      num #instances #bytes class name
      ----------------------------------------------
      1: 186449630 19326123016 [Ljava.lang.Object;
      2: 157366649 15107198304 org.apache.hadoop.hdfs.server.namenode.INodeFile
      3: 183409030 11738177920 org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo
      4: 157358401 5244264024 [Lorg.apache.hadoop.hdfs.server.blockmanagement.BlockInfo;
      5: 3 3489661000 [Lorg.apache.hadoop.util.LightWeightGSet$LinkedElement;
      6: 29253275 1872719664 [B
      7: 3230821 284312248 org.apache.hadoop.hdfs.server.namenode.INodeDirectory
      8: 2756284 110251360 java.util.ArrayList
      9: 469158 22519584 org.apache.hadoop.fs.permission.AclEntry
      10: 847 17133032 [Ljava.util.HashMap$Entry;
      11: 188471 17059632 [C
      12: 314614 10067656 [Lorg.apache.hadoop.hdfs.server.namenode.INode$Feature;
      13: 234579 9383160 com.google.common.collect.RegularImmutableList
      14: 49584 6850280 <constMethodKlass>
      15: 49584 6356704 <methodKlass>
      16: 187270 5992640 java.lang.String
      17: 234579 5629896 org.apache.hadoop.hdfs.server.namenode.AclFeature

      histo information at 2014-12-02 01:32

      num #instances #bytes class name
      ----------------------------------------------
      1: 355838051 35566651032 [Ljava.lang.Object;
      2: 302272758 29018184768 org.apache.hadoop.hdfs.server.namenode.INodeFile
      3: 352500723 22560046272 org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo
      4: 302264510 10075087952 [Lorg.apache.hadoop.hdfs.server.blockmanagement.BlockInfo;
      5: 177120233 9374983920 [B
      6: 3 3489661000 [Lorg.apache.hadoop.util.LightWeightGSet$LinkedElement;
      7: 6191688 544868544 org.apache.hadoop.hdfs.server.namenode.INodeDirectory
      8: 2799256 111970240 java.util.ArrayList
      9: 890728 42754944 org.apache.hadoop.fs.permission.AclEntry
      10: 330986 29974408 [C
      11: 596871 19099880 [Lorg.apache.hadoop.hdfs.server.namenode.INode$Feature;
      12: 445364 17814560 com.google.common.collect.RegularImmutableList
      13: 844 17132816 [Ljava.util.HashMap$Entry;
      14: 445364 10688736 org.apache.hadoop.hdfs.server.namenode.AclFeature
      15: 329789 10553248 java.lang.String
      16: 91741 8807136 org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoUnderConstruction
      17: 49584 6850280 <constMethodKlass>

      And the stack trace shows it was doing reloadFromImageFile:

      at org.apache.hadoop.hdfs.server.namenode.FSDirectory.getInode(FSDirectory.java:2426)
      at org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.loadINodeDirectorySection(FSImageFormatPBINode.java:160)
      at org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.loadInternal(FSImageFormatProtobuf.java:243)
      at org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.load(FSImageFormatProtobuf.java:168)
      at org.apache.hadoop.hdfs.server.namenode.FSImageFormat$LoaderDelegator.load(FSImageFormat.java:121)
      at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:902)
      at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:888)
      at org.apache.hadoop.hdfs.server.namenode.FSImage.reloadFromImageFile(FSImage.java:562)
      at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doMerge(SecondaryNameNode.java:1048)
      at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:536)
      at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:388)
      at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$1.run(SecondaryNameNode.java:354)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:356)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1630)
      at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:413)
      at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:350)
      at java.lang.Thread.run(Thread.java:745)

      So before doing reloadFromImageFile, I think we need release old namesystem to prevent SecondaryNameNode OOM.

      1. HDFS-7470.1.patch
        3 kB
        yunjiong zhao
      2. HDFS-7470.2.patch
        2 kB
        yunjiong zhao
      3. HDFS-7470.patch
        2 kB
        yunjiong zhao
      4. secondaryNameNode.jstack.txt
        14 kB
        yunjiong zhao

        Activity

        Hide
        zhaoyunjiong yunjiong zhao added a comment -

        This patch re-init namesystem before doing reloadFromImageFile.

        Show
        zhaoyunjiong yunjiong zhao added a comment - This patch re-init namesystem before doing reloadFromImageFile.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12684821/HDFS-7470.patch
        against trunk revision 7caa3bc.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.hdfs.server.balancer.TestBalancer
        org.apache.hadoop.hdfs.server.namenode.snapshot.TestCheckpointsWithSnapshots

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8909//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8909//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12684821/HDFS-7470.patch against trunk revision 7caa3bc. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.balancer.TestBalancer org.apache.hadoop.hdfs.server.namenode.snapshot.TestCheckpointsWithSnapshots +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8909//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8909//console This message is automatically generated.
        Hide
        zhaoyunjiong yunjiong zhao added a comment -

        Update patch to fix test failure.

        Show
        zhaoyunjiong yunjiong zhao added a comment - Update patch to fix test failure.
        Hide
        cnauroth Chris Nauroth added a comment -

        Creating a new FSNamesystem instance without running the full shutdown sequence on the old one would create a risk of some dangerous side effects.

        • A new namesystem lock instance would get created, and there would be no synchronization of multiple threads around this. This could violate mutual exclusion. Two different threads could hold the 2 different lock instances, and think that mutual exclusion has been enforced.
        • We wouldn't reap background threads inside things like the BlockManager and CacheManager. Over time, we'd slowly leak threads and eventually hit OutOfMemoryError conditions.
        • I can't remember if we hold an open file descriptor on the edit log when running as SecondaryNameNode. If we do, then discarding the old FSNamesystem without a proper shutdown would leak a file descriptor.

        In general, there are widespread assumptions throughout the codebase that FSNamesystem is instantiated exactly once and retained for the entire process lifetime.

        Show
        cnauroth Chris Nauroth added a comment - Creating a new FSNamesystem instance without running the full shutdown sequence on the old one would create a risk of some dangerous side effects. A new namesystem lock instance would get created, and there would be no synchronization of multiple threads around this. This could violate mutual exclusion. Two different threads could hold the 2 different lock instances, and think that mutual exclusion has been enforced. We wouldn't reap background threads inside things like the BlockManager and CacheManager . Over time, we'd slowly leak threads and eventually hit OutOfMemoryError conditions. I can't remember if we hold an open file descriptor on the edit log when running as SecondaryNameNode. If we do, then discarding the old FSNamesystem without a proper shutdown would leak a file descriptor. In general, there are widespread assumptions throughout the codebase that FSNamesystem is instantiated exactly once and retained for the entire process lifetime.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12685066/HDFS-7470.1.patch
        against trunk revision f6f2a3f.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        -1 findbugs. The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9017//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/9017//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9017//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12685066/HDFS-7470.1.patch against trunk revision f6f2a3f. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9017//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/9017//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9017//console This message is automatically generated.
        Hide
        zhaoyunjiong yunjiong zhao added a comment -

        Thanks Chris Nauroth for your time.
        Upload a stack trace file for SecondaryNameNode.

        Correct me if I'm wrong, from stack trace, I think there won't have two threads hold FSNamesystem.writeLock.
        And SecondaryNameNode didn't start service like BlockManager and CacheManager.
        For the edit log, SecondaryNameNode won't open it for write.

        I'll check again whether I missed some risk or try to find out a more safer solution later.

        Show
        zhaoyunjiong yunjiong zhao added a comment - Thanks Chris Nauroth for your time. Upload a stack trace file for SecondaryNameNode. Correct me if I'm wrong, from stack trace, I think there won't have two threads hold FSNamesystem.writeLock. And SecondaryNameNode didn't start service like BlockManager and CacheManager. For the edit log, SecondaryNameNode won't open it for write. I'll check again whether I missed some risk or try to find out a more safer solution later.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12686776/secondaryNameNode.jstack.txt
        against trunk revision b437f5e.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9020//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12686776/secondaryNameNode.jstack.txt against trunk revision b437f5e. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9020//console This message is automatically generated.
        Hide
        zhaoyunjiong yunjiong zhao added a comment -

        This patch will clear BlocksMap in FSNamesystem.clear().
        I believe this should release the memory.

        Show
        zhaoyunjiong yunjiong zhao added a comment - This patch will clear BlocksMap in FSNamesystem.clear(). I believe this should release the memory.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12691879/HDFS-7470.2.patch
        against trunk revision c4cba61.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.hdfs.TestHFlush

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9196//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9196//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12691879/HDFS-7470.2.patch against trunk revision c4cba61. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestHFlush Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9196//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9196//console This message is automatically generated.
        Hide
        cnauroth Chris Nauroth added a comment -

        +1 for the patch. This patch looks safer than the earlier version. yunjiong zhao, thank you for the contribution. I committed this to trunk and branch-2.

        Show
        cnauroth Chris Nauroth added a comment - +1 for the patch. This patch looks safer than the earlier version. yunjiong zhao , thank you for the contribution. I committed this to trunk and branch-2.
        Hide
        zhaoyunjiong yunjiong zhao added a comment -

        Chris, thanks for your time.

        Show
        zhaoyunjiong yunjiong zhao added a comment - Chris, thanks for your time.
        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        Pulled this into 2.6.1. The patch applied cleanly. Ran compilation before the push.

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - Pulled this into 2.6.1. The patch applied cleanly. Ran compilation before the push.
        Hide
        andrew.wang Andrew Wang added a comment -

        For future git log greppers, the JIRA ID was typo'd as HDFS-7570.

        Show
        andrew.wang Andrew Wang added a comment - For future git log greppers, the JIRA ID was typo'd as HDFS-7570 .

          People

          • Assignee:
            zhaoyunjiong yunjiong zhao
            Reporter:
            zhaoyunjiong yunjiong zhao
          • Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development