Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11209

SNN can't checkpoint when rolling upgrade is not finalized

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.8.0, 3.0.0-alpha1
    • Fix Version/s: 2.9.0, 3.0.0-alpha2
    • Component/s: rolling upgrades
    • Labels:
      None

      Description

      Similar problem has been fixed with HDFS-7185. Recent change in HDFS-8432 brings this back.

      With HDFS-8432, the primary NN will not update the VERSION file to the new version after running with "rollingUpgrade" option until upgrade is finalized. This is to support more downgrade use cases.

      However, the checkpoint on the SNN is incorrectly updating the VERSION file when the rollingUpgrade is not finalized yet on the primary NN. As a result, the SNN checkpoint successfully but fail to push it to the primary NN because its version is higher than the primary NN as shown below.

      2016-12-02 05:25:31,918 ERROR namenode.SecondaryNameNode (SecondaryNameNode.java:doWork(399)) - Exception in doCheckpoint
      org.apache.hadoop.hdfs.server.namenode.TransferFsImage$HttpPutFailedException: Image uploading failed, status: 403, url: http://NN:50070/imagetransfer?txid=345404754&imageFile=IMAGE&File-Le..., message: This namenode has storage info -60:221856466:1444080250181:clusterX but the secondary expected -63:221856466:1444080250181:clusterX
      
      1. HDFS-11209.00.patch
        7 kB
        Xiaoyu Yao
      2. HDFS-11209.01.patch
        9 kB
        Xiaoyu Yao
      3. HDFS-11209.02.patch
        10 kB
        Xiaoyu Yao
      4. HDFS-11209.03.patch
        11 kB
        Xiaoyu Yao
      5. HDFS-11209.04.patch
        10 kB
        Xiaoyu Yao

        Issue Links

          Activity

          Hide
          andrew.wang Andrew Wang added a comment -

          Thanks for the report Xiaoyu Yao, I'm setting the target/affects versions based on HDFS-8432.

          Since most users are in an HA setup these days, this might not be a blocker, but I think it's at least a critical issue.

          Show
          andrew.wang Andrew Wang added a comment - Thanks for the report Xiaoyu Yao , I'm setting the target/affects versions based on HDFS-8432 . Since most users are in an HA setup these days, this might not be a blocker, but I think it's at least a critical issue.
          Hide
          djp Junping Du added a comment -

          Hi Xiaoyu Yao, do we plan to fix this issue in short term? If not, please drop 2.8 with 2.9 instead. Thanks!

          Show
          djp Junping Du added a comment - Hi Xiaoyu Yao , do we plan to fix this issue in short term? If not, please drop 2.8 with 2.9 instead. Thanks!
          Hide
          djp Junping Du added a comment -

          No progress on this JIRA for a while. move to 2.9

          Show
          djp Junping Du added a comment - No progress on this JIRA for a while. move to 2.9
          Hide
          xyao Xiaoyu Yao added a comment -

          Thanks Junping Du for the heads up. Move to 2.9 sounds good to me.

          Show
          xyao Xiaoyu Yao added a comment - Thanks Junping Du for the heads up. Move to 2.9 sounds good to me.
          Hide
          xyao Xiaoyu Yao added a comment -

          Attach a patch to check the NN rolling upgrade status before update the VERSION file on SNN and Backup NN. The original code that checks the SNN namesystem rollingUpgrade won't work as SNN will never start with RollingUpgrade option. Backup NN should have the similar issue.

          Will add a unit test later.

          Show
          xyao Xiaoyu Yao added a comment - Attach a patch to check the NN rolling upgrade status before update the VERSION file on SNN and Backup NN. The original code that checks the SNN namesystem rollingUpgrade won't work as SNN will never start with RollingUpgrade option. Backup NN should have the similar issue. Will add a unit test later.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 13m 14s trunk passed
          +1 compile 0m 46s trunk passed
          +1 checkstyle 0m 28s trunk passed
          +1 mvnsite 0m 50s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 42s trunk passed
          +1 javadoc 0m 39s trunk passed
          -1 mvninstall 0m 27s hadoop-hdfs in the patch failed.
          -1 compile 0m 27s hadoop-hdfs in the patch failed.
          -1 cc 0m 27s hadoop-hdfs in the patch failed.
          -1 javac 0m 27s hadoop-hdfs in the patch failed.
          -0 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 149 unchanged - 0 fixed = 151 total (was 149)
          -1 mvnsite 0m 27s hadoop-hdfs in the patch failed.
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          -1 findbugs 0m 15s hadoop-hdfs in the patch failed.
          +1 javadoc 0m 35s the patch passed
          -1 unit 0m 29s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          22m 46s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-11209
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12846388/HDFS-11209.00.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux 4eb08e4769c8 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 91bf504
          Default Java 1.8.0_111
          findbugs v3.0.0
          mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/18110/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt
          compile https://builds.apache.org/job/PreCommit-HDFS-Build/18110/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
          cc https://builds.apache.org/job/PreCommit-HDFS-Build/18110/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
          javac https://builds.apache.org/job/PreCommit-HDFS-Build/18110/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18110/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          mvnsite https://builds.apache.org/job/PreCommit-HDFS-Build/18110/artifact/patchprocess/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/18110/artifact/patchprocess/patch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18110/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18110/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18110/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 13m 14s trunk passed +1 compile 0m 46s trunk passed +1 checkstyle 0m 28s trunk passed +1 mvnsite 0m 50s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 42s trunk passed +1 javadoc 0m 39s trunk passed -1 mvninstall 0m 27s hadoop-hdfs in the patch failed. -1 compile 0m 27s hadoop-hdfs in the patch failed. -1 cc 0m 27s hadoop-hdfs in the patch failed. -1 javac 0m 27s hadoop-hdfs in the patch failed. -0 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 149 unchanged - 0 fixed = 151 total (was 149) -1 mvnsite 0m 27s hadoop-hdfs in the patch failed. +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. -1 findbugs 0m 15s hadoop-hdfs in the patch failed. +1 javadoc 0m 35s the patch passed -1 unit 0m 29s hadoop-hdfs in the patch failed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 22m 46s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11209 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12846388/HDFS-11209.00.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 4eb08e4769c8 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 91bf504 Default Java 1.8.0_111 findbugs v3.0.0 mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/18110/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt compile https://builds.apache.org/job/PreCommit-HDFS-Build/18110/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt cc https://builds.apache.org/job/PreCommit-HDFS-Build/18110/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt javac https://builds.apache.org/job/PreCommit-HDFS-Build/18110/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18110/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt mvnsite https://builds.apache.org/job/PreCommit-HDFS-Build/18110/artifact/patchprocess/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/18110/artifact/patchprocess/patch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/18110/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18110/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18110/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xyao Xiaoyu Yao added a comment -

          Fix the build issue.

          Show
          xyao Xiaoyu Yao added a comment - Fix the build issue.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 14m 24s trunk passed
          +1 compile 0m 57s trunk passed
          +1 checkstyle 0m 33s trunk passed
          +1 mvnsite 1m 2s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 1m 54s trunk passed
          +1 javadoc 0m 43s trunk passed
          +1 mvninstall 0m 56s the patch passed
          +1 compile 0m 48s the patch passed
          +1 cc 0m 48s the patch passed
          +1 javac 0m 48s the patch passed
          -0 checkstyle 0m 28s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 151 unchanged - 0 fixed = 153 total (was 151)
          +1 mvnsite 0m 54s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 8s the patch passed
          +1 javadoc 0m 38s the patch passed
          -1 unit 76m 0s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 22s The patch does not generate ASF License warnings.
          103m 52s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeUUID
            hadoop.hdfs.server.namenode.web.resources.TestWebHdfsDataLocality
            hadoop.hdfs.server.datanode.checker.TestThrottledAsyncChecker
            hadoop.hdfs.server.datanode.TestDirectoryScanner
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-11209
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12846397/HDFS-11209.01.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux aee662cccc56 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 91bf504
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18112/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18112/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18112/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18112/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 14m 24s trunk passed +1 compile 0m 57s trunk passed +1 checkstyle 0m 33s trunk passed +1 mvnsite 1m 2s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 54s trunk passed +1 javadoc 0m 43s trunk passed +1 mvninstall 0m 56s the patch passed +1 compile 0m 48s the patch passed +1 cc 0m 48s the patch passed +1 javac 0m 48s the patch passed -0 checkstyle 0m 28s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 151 unchanged - 0 fixed = 153 total (was 151) +1 mvnsite 0m 54s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 8s the patch passed +1 javadoc 0m 38s the patch passed -1 unit 76m 0s hadoop-hdfs in the patch failed. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 103m 52s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeUUID   hadoop.hdfs.server.namenode.web.resources.TestWebHdfsDataLocality   hadoop.hdfs.server.datanode.checker.TestThrottledAsyncChecker   hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11209 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12846397/HDFS-11209.01.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux aee662cccc56 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 91bf504 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18112/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/18112/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18112/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18112/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xyao Xiaoyu Yao added a comment -

          Fix the checkstyle and adding unit test.

          Show
          xyao Xiaoyu Yao added a comment - Fix the checkstyle and adding unit test.
          Hide
          xyao Xiaoyu Yao added a comment -

          I've also manually tested on a cluster with rolling upgrade (non-HA) from NN layout version 60 to 63 and verify the patch fixed the checkpoint problem on SNN.

          Show
          xyao Xiaoyu Yao added a comment - I've also manually tested on a cluster with rolling upgrade (non-HA) from NN layout version 60 to 63 and verify the patch fixed the checkpoint problem on SNN.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 12m 38s trunk passed
          +1 compile 0m 46s trunk passed
          +1 checkstyle 0m 28s trunk passed
          +1 mvnsite 0m 49s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 1m 40s trunk passed
          +1 javadoc 0m 38s trunk passed
          +1 mvninstall 0m 45s the patch passed
          +1 compile 0m 44s the patch passed
          +1 cc 0m 44s the patch passed
          +1 javac 0m 44s the patch passed
          +1 checkstyle 0m 26s the patch passed
          +1 mvnsite 0m 47s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 46s the patch passed
          +1 javadoc 0m 37s the patch passed
          -1 unit 68m 26s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 21s The patch does not generate ASF License warnings.
          92m 36s



          Reason Tests
          Timed out junit tests org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-11209
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12846426/HDFS-11209.02.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux 7f8814d53b92 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 91bf504
          Default Java 1.8.0_111
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18118/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18118/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18118/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 12m 38s trunk passed +1 compile 0m 46s trunk passed +1 checkstyle 0m 28s trunk passed +1 mvnsite 0m 49s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 40s trunk passed +1 javadoc 0m 38s trunk passed +1 mvninstall 0m 45s the patch passed +1 compile 0m 44s the patch passed +1 cc 0m 44s the patch passed +1 javac 0m 44s the patch passed +1 checkstyle 0m 26s the patch passed +1 mvnsite 0m 47s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 46s the patch passed +1 javadoc 0m 37s the patch passed -1 unit 68m 26s hadoop-hdfs in the patch failed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 92m 36s Reason Tests Timed out junit tests org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11209 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12846426/HDFS-11209.02.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 7f8814d53b92 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 91bf504 Default Java 1.8.0_111 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/18118/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18118/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18118/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xyao Xiaoyu Yao added a comment - - edited

          Update patch for HDFS-11209.03.patch. Diff from prev: consolidate the NN layout version update logic into FSImage#updateStorageVersion() to avoid future bugs in this area.

          Show
          xyao Xiaoyu Yao added a comment - - edited Update patch for HDFS-11209 .03.patch. Diff from prev: consolidate the NN layout version update logic into FSImage#updateStorageVersion() to avoid future bugs in this area.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 13m 14s trunk passed
          +1 compile 0m 45s trunk passed
          +1 checkstyle 0m 28s trunk passed
          +1 mvnsite 0m 52s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 1m 50s trunk passed
          +1 javadoc 0m 39s trunk passed
          +1 mvninstall 0m 50s the patch passed
          +1 compile 0m 52s the patch passed
          +1 cc 0m 52s the patch passed
          +1 javac 0m 52s the patch passed
          +1 checkstyle 0m 28s the patch passed
          +1 mvnsite 0m 53s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 3s the patch passed
          +1 javadoc 0m 41s the patch passed
          -1 unit 81m 19s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 24s The patch does not generate ASF License warnings.
          107m 24s



          Reason Tests
          Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure020
          Timed out junit tests org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-11209
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12846710/HDFS-11209.03.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux dc6f9d09920a 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / e692316
          Default Java 1.8.0_111
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18135/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18135/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18135/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 13m 14s trunk passed +1 compile 0m 45s trunk passed +1 checkstyle 0m 28s trunk passed +1 mvnsite 0m 52s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 50s trunk passed +1 javadoc 0m 39s trunk passed +1 mvninstall 0m 50s the patch passed +1 compile 0m 52s the patch passed +1 cc 0m 52s the patch passed +1 javac 0m 52s the patch passed +1 checkstyle 0m 28s the patch passed +1 mvnsite 0m 53s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 3s the patch passed +1 javadoc 0m 41s the patch passed -1 unit 81m 19s hadoop-hdfs in the patch failed. +1 asflicense 0m 24s The patch does not generate ASF License warnings. 107m 24s Reason Tests Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure020 Timed out junit tests org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11209 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12846710/HDFS-11209.03.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux dc6f9d09920a 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e692316 Default Java 1.8.0_111 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/18135/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18135/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18135/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xyao Xiaoyu Yao added a comment - - edited

          Delta from v03: removing the unit test change that can't repro the original rolling upgrade issue.

          The repro is a bit tricky with MiniDFSCluter as we need to run old version of NN with ""hdfs dfsadmin -rollingUpgrade prepare" to create a fsiamge with the old layoutversion. Then do the upgrade and run the primary namenode(new software layout version) with "-rollingUpgrade started" option and secondary namenode (new software layout version) as normal.

          The software layout version is determined by static method from LayoutVersion class which is not supported with mockito. It is possible to do that with powermock + mockito. Decide to add unit test in a separate ticket. I've manually tested upgrade from Hadoop 2.6 to Hadoop 2.7.1 in a non-HA setup with layout version changing 60->63 and verified that the SNN can checkpoint with an unfinalized primary NN rollingupgrade.

          Show
          xyao Xiaoyu Yao added a comment - - edited Delta from v03: removing the unit test change that can't repro the original rolling upgrade issue. The repro is a bit tricky with MiniDFSCluter as we need to run old version of NN with ""hdfs dfsadmin -rollingUpgrade prepare" to create a fsiamge with the old layoutversion. Then do the upgrade and run the primary namenode(new software layout version) with "-rollingUpgrade started" option and secondary namenode (new software layout version) as normal. The software layout version is determined by static method from LayoutVersion class which is not supported with mockito. It is possible to do that with powermock + mockito. Decide to add unit test in a separate ticket. I've manually tested upgrade from Hadoop 2.6 to Hadoop 2.7.1 in a non-HA setup with layout version changing 60->63 and verified that the SNN can checkpoint with an unfinalized primary NN rollingupgrade.
          Hide
          xyao Xiaoyu Yao added a comment -

          Opened HDFS-11341 for the unit test if possible.

          Show
          xyao Xiaoyu Yao added a comment - Opened HDFS-11341 for the unit test if possible.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 15m 37s trunk passed
          +1 compile 0m 59s trunk passed
          +1 checkstyle 0m 31s trunk passed
          +1 mvnsite 1m 1s trunk passed
          +1 mvneclipse 0m 17s trunk passed
          +1 findbugs 1m 54s trunk passed
          +1 javadoc 0m 43s trunk passed
          +1 mvninstall 0m 54s the patch passed
          +1 compile 0m 49s the patch passed
          +1 cc 0m 49s the patch passed
          +1 javac 0m 49s the patch passed
          +1 checkstyle 0m 28s the patch passed
          +1 mvnsite 0m 55s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 6s the patch passed
          +1 javadoc 0m 40s the patch passed
          -1 unit 93m 54s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 26s The patch does not generate ASF License warnings.
          123m 4s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.namenode.ha.TestHASafeMode
            hadoop.hdfs.server.datanode.checker.TestThrottledAsyncChecker
            hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation
            hadoop.hdfs.server.datanode.TestDataNodeUUID
          Timed out junit tests org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
            org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean
            org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-11209
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12847435/HDFS-11209.04.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux c0a7b8f5b306 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / d3170f9
          Default Java 1.8.0_111
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18173/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18173/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18173/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 15m 37s trunk passed +1 compile 0m 59s trunk passed +1 checkstyle 0m 31s trunk passed +1 mvnsite 1m 1s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 1m 54s trunk passed +1 javadoc 0m 43s trunk passed +1 mvninstall 0m 54s the patch passed +1 compile 0m 49s the patch passed +1 cc 0m 49s the patch passed +1 javac 0m 49s the patch passed +1 checkstyle 0m 28s the patch passed +1 mvnsite 0m 55s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 6s the patch passed +1 javadoc 0m 40s the patch passed -1 unit 93m 54s hadoop-hdfs in the patch failed. +1 asflicense 0m 26s The patch does not generate ASF License warnings. 123m 4s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.ha.TestHASafeMode   hadoop.hdfs.server.datanode.checker.TestThrottledAsyncChecker   hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation   hadoop.hdfs.server.datanode.TestDataNodeUUID Timed out junit tests org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean   org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11209 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12847435/HDFS-11209.04.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux c0a7b8f5b306 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d3170f9 Default Java 1.8.0_111 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/18173/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18173/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18173/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          +1 for the v04 patch assuming the test failures are unrelated.

          One minor point - the isRollingUpgrade RPC need not check for super user privilege since it's harmless. Doesn't affect correctness though since the SNN would be running as the hdfs superuser.

          Show
          arpitagarwal Arpit Agarwal added a comment - +1 for the v04 patch assuming the test failures are unrelated. One minor point - the isRollingUpgrade RPC need not check for super user privilege since it's harmless. Doesn't affect correctness though since the SNN would be running as the hdfs superuser.
          Hide
          xyao Xiaoyu Yao added a comment -

          Thanks Arpit Agarwal for the review. It is a good point that we may allow isRollingUpgrade() from non hdfs superuse even though the current usage is for hdfs superuser only with SNN.

          This applies to a similar API NameNodeRpcServer#isUpgradeFinalized(). I will open a separate ticket discussing whether we should remove the super user privilege check for both.

          Show
          xyao Xiaoyu Yao added a comment - Thanks Arpit Agarwal for the review. It is a good point that we may allow isRollingUpgrade() from non hdfs superuse even though the current usage is for hdfs superuser only with SNN. This applies to a similar API NameNodeRpcServer#isUpgradeFinalized(). I will open a separate ticket discussing whether we should remove the super user privilege check for both.
          Hide
          xyao Xiaoyu Yao added a comment -

          Junping Du, can we add this one to 2.8 as checkpoint is critical to SNN?

          Show
          xyao Xiaoyu Yao added a comment - Junping Du , can we add this one to 2.8 as checkpoint is critical to SNN?
          Hide
          xyao Xiaoyu Yao added a comment -

          Thanks Arpit Agarwal for the review. I've commit the fix to trunk and branch-2.

          Show
          xyao Xiaoyu Yao added a comment - Thanks Arpit Agarwal for the review. I've commit the fix to trunk and branch-2.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #11130 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11130/)
          HDFS-11209. SNN can't checkpoint when rolling upgrade is not finalized. (xyao: rev b1fce2b8b14c4ce43f7098a269ac2b95304db0ce)

          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/proto/NamenodeProtocol.proto
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Checkpointer.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/NamenodeProtocol.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolServerSideTranslatorPB.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolTranslatorPB.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #11130 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11130/ ) HDFS-11209 . SNN can't checkpoint when rolling upgrade is not finalized. (xyao: rev b1fce2b8b14c4ce43f7098a269ac2b95304db0ce) (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/proto/NamenodeProtocol.proto (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Checkpointer.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/NamenodeProtocol.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolServerSideTranslatorPB.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolTranslatorPB.java
          Hide
          kanaka Kanaka Kumar Avvaru added a comment -

          Xiaoyu Yao, I think this issue is applicable to branch 2.8 also. Can you please commit to 2.8 also?
          Ping Junping Du, Arpit Agarwal

          Show
          kanaka Kanaka Kumar Avvaru added a comment - Xiaoyu Yao , I think this issue is applicable to branch 2.8 also. Can you please commit to 2.8 also? Ping Junping Du , Arpit Agarwal

            People

            • Assignee:
              xyao Xiaoyu Yao
              Reporter:
              xyao Xiaoyu Yao
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development