Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-5794

Fix the inconsistency of layout version number of ADD_DATANODE_AND_STORAGE_UUIDS between trunk and branch-2

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 3.0.0, 2.3.0
    • Fix Version/s: 3.0.0
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently in trunk, we have the layout version:

      EDITLOG_ADD_BLOCK(-48, ...),
      CACHING(-49, ...),
      ADD_DATANODE_AND_STORAGE_UUIDS(-50, ...);
      

      And in branch-2, we have:

      EDITLOG_SUPPORT_RETRYCACHE(-47, ...),
      ADD_DATANODE_AND_STORAGE_UUIDS(-49, -47, ...);
      

      We plan to backport HDFS-5704 and HDFS-5777 to branch-2, thus EDITLOG_ADD_BLOCK will also take -48 in branch-2. However, we cannot change ADD_DATANODE_AND_STORAGE_UUIDS to -50 in branch-2. Otherwise fsimages written by trunk and branch-2 have the same layout -50 but branch-2 cannot read the -50 fsimage if it is written by trunk.

      1. HDFS-5794.001.patch
        1 kB
        Jing Zhao
      2. HDFS-5794.000.patch
        1 kB
        Jing Zhao

        Activity

        Jing Zhao created issue -
        Jing Zhao made changes -
        Field Original Value New Value
        Description Currently in trunk, we have the layout version:
        {code}
            EDITLOG_ADD_BLOCK(-48, ...),
            CACHING(-49, ...),
            ADD_DATANODE_AND_STORAGE_UUIDS(-50, ...);
        {code}

        And in branch-2, we have:
        {code}
        EDITLOG_SUPPORT_RETRYCACHE(-47, ...),
        ADD_DATANODE_AND_STORAGE_UUIDS(-49, -47, ...);
        {code}

        We plan to backport HDFS-5704 and HDFS-5777 to branch-2, thus EDITLOG_ADD_BLOCK will also take -48 in branch-2. However, we cannot change ADD_DATANODE_AND_STORAGE_UUIDS to -50 in branch-2. Otherwise fsimages written by trunk and branch-2 have the same layout -50 but branch-2 cannot read the -50 fsimage if it is written by trunk.
        Currently in trunk, we have the layout version:
        {code}
        EDITLOG_ADD_BLOCK(-48, ...),
        CACHING(-49, ...),
        ADD_DATANODE_AND_STORAGE_UUIDS(-50, ...);
        {code}

        And in branch-2, we have:
        {code}
        EDITLOG_SUPPORT_RETRYCACHE(-47, ...),
        ADD_DATANODE_AND_STORAGE_UUIDS(-49, -47, ...);
        {code}

        We plan to backport HDFS-5704 and HDFS-5777 to branch-2, thus EDITLOG_ADD_BLOCK will also take -48 in branch-2. However, we cannot change ADD_DATANODE_AND_STORAGE_UUIDS to -50 in branch-2. Otherwise fsimages written by trunk and branch-2 have the same layout -50 but branch-2 cannot read the -50 fsimage if it is written by trunk.
        Hide
        Jing Zhao added a comment -

        One solution for this issue is to switch CACHING and ADD_DATANODE_AND_STORAGE_UUIDS in trunk. Post a simple patch following this solution.

        Show
        Jing Zhao added a comment - One solution for this issue is to switch CACHING and ADD_DATANODE_AND_STORAGE_UUIDS in trunk. Post a simple patch following this solution.
        Jing Zhao made changes -
        Attachment HDFS-5794.000.patch [ 12623524 ]
        Jing Zhao made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Colin Patrick McCabe added a comment -

        +1 for this approach of putting CACHING last, once you've rebased and re-run jenkins. HDFS-5784 added another layout version which CACHING should come after.

        Show
        Colin Patrick McCabe added a comment - +1 for this approach of putting CACHING last, once you've rebased and re-run jenkins. HDFS-5784 added another layout version which CACHING should come after.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12623524/HDFS-5794.000.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/5905//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5905//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12623524/HDFS-5794.000.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/5905//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5905//console This message is automatically generated.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        +1 patch looks good.

        Show
        Tsz Wo Nicholas Sze added a comment - +1 patch looks good.
        Tsz Wo Nicholas Sze made changes -
        Priority Major [ 3 ] Minor [ 4 ]
        Component/s namenode [ 12312926 ]
        Hide
        Jing Zhao added a comment -

        Thanks for the review, Nicholas and Colin! Rebase the patch and put CACHING last.

        Show
        Jing Zhao added a comment - Thanks for the review, Nicholas and Colin! Rebase the patch and put CACHING last.
        Jing Zhao made changes -
        Attachment HDFS-5794.001.patch [ 12623592 ]
        Hide
        Tsz Wo Nicholas Sze added a comment -

        +1 the new patch looks good.

        Show
        Tsz Wo Nicholas Sze added a comment - +1 the new patch looks good.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12623592/HDFS-5794.001.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/5909//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5909//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12623592/HDFS-5794.001.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/5909//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5909//console This message is automatically generated.
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Hadoop-trunk-Commit #5018 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5018/)
        HDFS-5794. Fix the inconsistency of layout version number of ADD_DATANODE_AND_STORAGE_UUIDS between trunk and branch-2. Contributed by Jing Zhao. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1559209)

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LayoutVersion.java
        Show
        Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #5018 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5018/ ) HDFS-5794 . Fix the inconsistency of layout version number of ADD_DATANODE_AND_STORAGE_UUIDS between trunk and branch-2. Contributed by Jing Zhao. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1559209 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LayoutVersion.java
        Hide
        Jing Zhao added a comment -

        I've committed this. The change is only needed in trunk. Layout version changes in branch-2 will later be covered by merging HDFS-5777.

        Show
        Jing Zhao added a comment - I've committed this. The change is only needed in trunk. Layout version changes in branch-2 will later be covered by merging HDFS-5777 .
        Jing Zhao made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags Reviewed [ 10343 ]
        Fix Version/s 3.0.0 [ 12320356 ]
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Yarn-trunk #456 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/456/)
        HDFS-5794. Fix the inconsistency of layout version number of ADD_DATANODE_AND_STORAGE_UUIDS between trunk and branch-2. Contributed by Jing Zhao. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1559209)

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LayoutVersion.java
        Show
        Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #456 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/456/ ) HDFS-5794 . Fix the inconsistency of layout version number of ADD_DATANODE_AND_STORAGE_UUIDS between trunk and branch-2. Contributed by Jing Zhao. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1559209 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LayoutVersion.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #1648 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1648/)
        HDFS-5794. Fix the inconsistency of layout version number of ADD_DATANODE_AND_STORAGE_UUIDS between trunk and branch-2. Contributed by Jing Zhao. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1559209)

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LayoutVersion.java
        Show
        Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #1648 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1648/ ) HDFS-5794 . Fix the inconsistency of layout version number of ADD_DATANODE_AND_STORAGE_UUIDS between trunk and branch-2. Contributed by Jing Zhao. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1559209 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LayoutVersion.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #1673 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1673/)
        HDFS-5794. Fix the inconsistency of layout version number of ADD_DATANODE_AND_STORAGE_UUIDS between trunk and branch-2. Contributed by Jing Zhao. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1559209)

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LayoutVersion.java
        Show
        Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1673 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1673/ ) HDFS-5794 . Fix the inconsistency of layout version number of ADD_DATANODE_AND_STORAGE_UUIDS between trunk and branch-2. Contributed by Jing Zhao. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1559209 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LayoutVersion.java
        Arun C Murthy made changes -
        Affects Version/s 2.3.0 [ 12325255 ]
        Affects Version/s 2.4.0 [ 12324588 ]

          People

          • Assignee:
            Jing Zhao
            Reporter:
            Jing Zhao
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development