Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-3415

During NameNode starting up, it may pick wrong storage directory inspector when the layout versions of the storage directories are different

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha, 3.0.0
    • Fix Version/s: 2.0.2-alpha
    • Component/s: namenode
    • Labels:
      None
    • Environment:

      Suse linux + jdk 1.6

      Description

      Scenario:
      =========
      start Namenode and datanode by configuring three storage dir's for namenode
      write 10 files
      edit version file of one of the storage dir and give layout version as 123 which different with default(-40).
      Stop namenode
      start Namenode.

      Then I am getting follwong exception...

      2012-05-13 19:01:41,483 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
      java.lang.NullPointerException
      	at org.apache.hadoop.hdfs.server.namenode.NNStorage.getStorageFile(NNStorage.java:686)
      	at org.apache.hadoop.hdfs.server.namenode.FSImagePreTransactionalStorageInspector.getEditsInStorageDir(FSImagePreTransactionalStorageInspector.java:243)
      	at org.apache.hadoop.hdfs.server.namenode.FSImagePreTransactionalStorageInspector.getLatestEditsFiles(FSImagePreTransactionalStorageInspector.java:261)
      	at org.apache.hadoop.hdfs.server.namenode.FSImagePreTransactionalStorageInspector.getEditLogStreams(FSImagePreTransactionalStorageInspector.java:276)
      	at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:596)
      	at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:247)
      	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:498)
      	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:390)
      	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:354)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:368)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:402)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:564)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:545)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1093)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1151)
      2012-05-13 19:01:41,485 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG: 
      
      
      1. HDFS-3415.patch
        4 kB
        Brandon Li
      2. HDFS-3415.patch
        4 kB
        Brandon Li
      3. HDFS-3415.patch
        3 kB
        Brandon Li
      4. HDFS-3415.patch
        2 kB
        Brandon Li

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1088 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1088/)
          HDFS-3415. Make sure all layout versions are the same for all storage directories in the Namenode. Contributed by Brandon Li (Revision 1341676)

          Result = SUCCESS
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1341676
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1088 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1088/ ) HDFS-3415 . Make sure all layout versions are the same for all storage directories in the Namenode. Contributed by Brandon Li (Revision 1341676) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1341676 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1054 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1054/)
          HDFS-3415. Make sure all layout versions are the same for all storage directories in the Namenode. Contributed by Brandon Li (Revision 1341676)

          Result = SUCCESS
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1341676
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1054 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1054/ ) HDFS-3415 . Make sure all layout versions are the same for all storage directories in the Namenode. Contributed by Brandon Li (Revision 1341676) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1341676 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2298 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2298/)
          HDFS-3415. Make sure all layout versions are the same for all storage directories in the Namenode. Contributed by Brandon Li (Revision 1341676)

          Result = FAILURE
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1341676
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2298 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2298/ ) HDFS-3415 . Make sure all layout versions are the same for all storage directories in the Namenode. Contributed by Brandon Li (Revision 1341676) Result = FAILURE szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1341676 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2280 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2280/)
          HDFS-3415. Make sure all layout versions are the same for all storage directories in the Namenode. Contributed by Brandon Li (Revision 1341676)

          Result = SUCCESS
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1341676
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2280 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2280/ ) HDFS-3415 . Make sure all layout versions are the same for all storage directories in the Namenode. Contributed by Brandon Li (Revision 1341676) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1341676 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2353 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2353/)
          HDFS-3415. Make sure all layout versions are the same for all storage directories in the Namenode. Contributed by Brandon Li (Revision 1341676)

          Result = SUCCESS
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1341676
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2353 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2353/ ) HDFS-3415 . Make sure all layout versions are the same for all storage directories in the Namenode. Contributed by Brandon Li (Revision 1341676) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1341676 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I have committed this. Thanks, Brandon!

          Show
          Tsz Wo Nicholas Sze added a comment - I have committed this. Thanks, Brandon!
          Hide
          Brandon Li added a comment -

          Manually tested it. It works as expected: throws exception if there are multiple layout versions.

          Show
          Brandon Li added a comment - Manually tested it. It works as expected: throws exception if there are multiple layout versions.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Hi Brandon, since there is no new tests, could you manually test it?

          Show
          Tsz Wo Nicholas Sze added a comment - Hi Brandon, since there is no new tests, could you manually test it?
          Hide
          Tsz Wo Nicholas Sze added a comment -

          +1 patch looks good.

          Show
          Tsz Wo Nicholas Sze added a comment - +1 patch looks good.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12528528/HDFS-3415.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2499//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2499//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12528528/HDFS-3415.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2499//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2499//console This message is automatically generated.
          Hide
          Brandon Li added a comment -

          Fix the warning.
          The test failures are unrelated.

          Show
          Brandon Li added a comment - Fix the warning. The test failures are unrelated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12528471/HDFS-3415.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.TestListFilesInFileContext
          org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks
          org.apache.hadoop.hdfs.server.datanode.TestBPOfferService
          org.apache.hadoop.hdfs.TestDatanodeBlockScanner

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2497//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/2497//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2497//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12528471/HDFS-3415.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestListFilesInFileContext org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks org.apache.hadoop.hdfs.server.datanode.TestBPOfferService org.apache.hadoop.hdfs.TestDatanodeBlockScanner +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2497//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/2497//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2497//console This message is automatically generated.
          Hide
          Brandon Li added a comment -
          {quota}I actually want you remove the min and max computation since it does not make sense to compute min and max for checking equality.{quota}

          Got it. minversion and maxversion are not needed anymore. Done.

          {quota}You should use StringBuilder instead of String as indicated by the new findbugs warning.{quota}

          Done. Thanks!

          Show
          Brandon Li added a comment - {quota}I actually want you remove the min and max computation since it does not make sense to compute min and max for checking equality.{quota} Got it. minversion and maxversion are not needed anymore. Done. {quota}You should use StringBuilder instead of String as indicated by the new findbugs warning.{quota} Done. Thanks!
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I actually want you remove the min and max computation since it does not make sense to compute min and max for checking equality. It is okay if you don't want to. However, please remove "if (minLayoutVersion > maxLayoutVersion)

          {..}

          ".

          You should use StringBuilder instead of String as indicated by the new findbugs warning.

          Show
          Tsz Wo Nicholas Sze added a comment - I actually want you remove the min and max computation since it does not make sense to compute min and max for checking equality. It is okay if you don't want to. However, please remove "if (minLayoutVersion > maxLayoutVersion) {..} ". You should use StringBuilder instead of String as indicated by the new findbugs warning .
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12528188/HDFS-3415.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 javadoc. The javadoc tool appears to have generated 2 warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2484//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/2484//artifact/trunk/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2484//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12528188/HDFS-3415.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 javadoc. The javadoc tool appears to have generated 2 warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2484//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/2484//artifact/trunk/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2484//console This message is automatically generated.
          Hide
          Brandon Li added a comment -

          We may first put all the dirs and versions in a map (Map<File, Integer>: root dir => version), check if all the versions are equal and, if not, print out all dirs and versions.

          Done with minor difference that the message is saved in a string not a map.

          Show
          Brandon Li added a comment - We may first put all the dirs and versions in a map (Map<File, Integer>: root dir => version), check if all the versions are equal and, if not, print out all dirs and versions. Done with minor difference that the message is saved in a string not a map.
          Hide
          Brandon Li added a comment -

          New improved the error information. The error print out like the following:

          java.io.IOException: Storage directories containe different layout versions: (/private/tmp/singlenn, -40) (/private/tmp/singlenn-edit, 123)

          Show
          Brandon Li added a comment - New improved the error information. The error print out like the following: java.io.IOException: Storage directories containe different layout versions: (/private/tmp/singlenn, -40) (/private/tmp/singlenn-edit, 123)
          Hide
          Tsz Wo Nicholas Sze added a comment -

          The patch looks good. Could you also improve the error message? We may first put all the dirs and versions in a map (Map<File, Integer>: root dir => version), check if all the versions are equal and, if not, print out all dirs and versions.

          Show
          Tsz Wo Nicholas Sze added a comment - The patch looks good. Could you also improve the error message? We may first put all the dirs and versions in a map (Map<File, Integer>: root dir => version), check if all the versions are equal and, if not, print out all dirs and versions.
          Hide
          Brandon Li added a comment -

          The warnings are not introduced by this patch. I did manually test to verify an exception is thrown when the layout versions are not the same.

          Show
          Brandon Li added a comment - The warnings are not introduced by this patch. I did manually test to verify an exception is thrown when the layout versions are not the same.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12527967/HDFS-3415.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 javadoc. The javadoc tool appears to have generated 2 warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2466//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2466//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12527967/HDFS-3415.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 javadoc. The javadoc tool appears to have generated 2 warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2466//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2466//console This message is automatically generated.
          Hide
          Brandon Li added a comment -

          Instead of allowing namenode to move forward with multiple layout versions, the storage inspector selector (NNStroage.readAndInspectDirs) should throw exception saying the inconsistent layout versions.

          Show
          Brandon Li added a comment - Instead of allowing namenode to move forward with multiple layout versions, the storage inspector selector (NNStroage.readAndInspectDirs) should throw exception saying the inconsistent layout versions.
          Hide
          Brandon Li added a comment -

          I reproduce this problem with the following configuration:
          1. set two storage directories, say dirA and dirB
          2. start and then shutdown namenode
          3. change only dirB's layout version from -40 to 123.
          4. start namenode and it should fail with the above NullPointerException

          The problem here is:

          Two storage inspectors are used in namenode, FSImagePreTransactionalStorageInspector for layout version before -38, and FSImageTransactionalStorageInspector for -38 or anything later.

          In this case, the modified storage directory happens to be the last one inspected by the namenode in order to load image/edits. Even though it sees two layout version, it saves the last one ("123" in this case) as the storage layout version. However, it uses FSImageTransactionalStorageInspector to get image path because dirA still has -40 and then uses FSImagePreTransactionalStorageInspector to get edit stream. Because FSImagePreTransactionalStorageInspector can't recognize
          the file in a storage directory whose real version is newer, some references are not initialized which eventually cause the exception.

          Show
          Brandon Li added a comment - I reproduce this problem with the following configuration: 1. set two storage directories, say dirA and dirB 2. start and then shutdown namenode 3. change only dirB's layout version from -40 to 123. 4. start namenode and it should fail with the above NullPointerException The problem here is: Two storage inspectors are used in namenode, FSImagePreTransactionalStorageInspector for layout version before -38, and FSImageTransactionalStorageInspector for -38 or anything later. In this case, the modified storage directory happens to be the last one inspected by the namenode in order to load image/edits. Even though it sees two layout version, it saves the last one ("123" in this case) as the storage layout version. However, it uses FSImageTransactionalStorageInspector to get image path because dirA still has -40 and then uses FSImagePreTransactionalStorageInspector to get edit stream. Because FSImagePreTransactionalStorageInspector can't recognize the file in a storage directory whose real version is newer, some references are not initialized which eventually cause the exception.

            People

            • Assignee:
              Brandon Li
              Reporter:
              Brahma Reddy Battula
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development