Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9470

Encryption zone on root not loaded from fsimage after NN restart

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.6.0
    • Fix Version/s: 2.8.0, 2.7.2, 2.6.3, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None

      Description

      When restarting namenode, the encryption zone for rootDir is not loaded correctly from fsimage

      1. HDFS-9470.001.patch
        4 kB
        Xiao Chen
      2. HDFS-9470.002.patch
        5 kB
        Xiao Chen
      3. HDFS-9470.003.patch
        5 kB
        Xiao Chen

        Activity

        Hide
        xiaochen Xiao Chen added a comment -

        Attached patch 1 fixes the issue. Locally verified the unit test fails without the fix, passes with.
        I choose to add the method addRootDirToEncryptionZone so that ezManager is still private in FSDirectory. (Directly using addToInodeMap would be problematic, because the INodeDirectory root is temporary and not the same object as FSDirectory.rootDir)

        Show
        xiaochen Xiao Chen added a comment - Attached patch 1 fixes the issue. Locally verified the unit test fails without the fix, passes with. I choose to add the method addRootDirToEncryptionZone so that ezManager is still private in FSDirectory . (Directly using addToInodeMap would be problematic, because the INodeDirectory root is temporary and not the same object as FSDirectory.rootDir )
        Hide
        andrew.wang Andrew Wang added a comment -

        Hi Xiao, thanks for picking this up, overall looks good. Only a few comments:

        • Is it possible to share some code between this new function and addToInodeMap? We could have a more generically named helper function, and call it from both addToInodeMap and loadRootInode. I don't think the log print is that important, can keep it generic.
        • Little nit, we already have the XAttrFeature in loadRootINode, so we could pass it instead of looking it up again inside addRootDirToEncryptionZone.
        Show
        andrew.wang Andrew Wang added a comment - Hi Xiao, thanks for picking this up, overall looks good. Only a few comments: Is it possible to share some code between this new function and addToInodeMap? We could have a more generically named helper function, and call it from both addToInodeMap and loadRootInode. I don't think the log print is that important, can keep it generic. Little nit, we already have the XAttrFeature in loadRootINode, so we could pass it instead of looking it up again inside addRootDirToEncryptionZone.
        Hide
        xiaochen Xiao Chen added a comment -

        Figured I should also ping Vinod Kumar Vavilapalli and Sangjin Lee like Andrew Wang did in HDFS-9273. This seems to be also a critical one.
        I'm still putting up fix #2 though.

        Show
        xiaochen Xiao Chen added a comment - Figured I should also ping Vinod Kumar Vavilapalli and Sangjin Lee like Andrew Wang did in HDFS-9273 . This seems to be also a critical one. I'm still putting up fix #2 though.
        Hide
        andrew.wang Andrew Wang added a comment -

        I went ahead and set the target versions accordingly, if the ACL fix goes in this one should too. Thx Xiao.

        Show
        andrew.wang Andrew Wang added a comment - I went ahead and set the target versions accordingly, if the ACL fix goes in this one should too. Thx Xiao.
        Hide
        xiaochen Xiao Chen added a comment -

        Thanks a lot Andrew Wang for the review and setting target versions.
        Your comments make sense to me. Attached patch 2 to refactor as you suggested.

        Show
        xiaochen Xiao Chen added a comment - Thanks a lot Andrew Wang for the review and setting target versions. Your comments make sense to me. Attached patch 2 to refactor as you suggested.
        Hide
        andrew.wang Andrew Wang added a comment -

        +1 LGTM pending jenkins

        Show
        andrew.wang Andrew Wang added a comment - +1 LGTM pending jenkins
        Hide
        xiaochen Xiao Chen added a comment -

        Thanks Andrew.
        Also answering the question asked in HDFS-9273: this one is also long standing, since the initial implementation of HDFS-6516.

        Show
        xiaochen Xiao Chen added a comment - Thanks Andrew. Also answering the question asked in HDFS-9273 : this one is also long standing, since the initial implementation of HDFS-6516 .
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 8m 0s trunk passed
        +1 compile 0m 42s trunk passed with JDK v1.8.0_66
        +1 compile 0m 44s trunk passed with JDK v1.7.0_85
        +1 checkstyle 0m 16s trunk passed
        +1 mvnsite 0m 52s trunk passed
        +1 mvneclipse 0m 14s trunk passed
        +1 findbugs 1m 59s trunk passed
        +1 javadoc 1m 10s trunk passed with JDK v1.8.0_66
        +1 javadoc 1m 52s trunk passed with JDK v1.7.0_85
        +1 mvninstall 0m 51s the patch passed
        +1 compile 0m 43s the patch passed with JDK v1.8.0_66
        +1 javac 0m 43s the patch passed
        +1 compile 0m 43s the patch passed with JDK v1.7.0_85
        +1 javac 0m 43s the patch passed
        -1 checkstyle 0m 16s Patch generated 1 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 66, now 67).
        +1 mvnsite 0m 55s the patch passed
        +1 mvneclipse 0m 14s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 2m 11s the patch passed
        +1 javadoc 1m 4s the patch passed with JDK v1.8.0_66
        +1 javadoc 1m 55s the patch passed with JDK v1.7.0_85
        -1 unit 52m 58s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
        +1 unit 49m 29s hadoop-hdfs in the patch passed with JDK v1.7.0_85.
        -1 asflicense 0m 20s Patch generated 58 ASF License warnings.
        130m 8s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes
          hadoop.hdfs.server.datanode.TestBlockScanner
          hadoop.hdfs.shortcircuit.TestShortCircuitCache



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12774450/HDFS-9470.002.patch
        JIRA Issue HDFS-9470
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux f25d2cde9159 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 0348e76
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13672/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13672/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13672/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
        JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13672/testReport/
        asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13672/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Max memory used 76MB
        Powered by Apache Yetus http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13672/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 0s trunk passed +1 compile 0m 42s trunk passed with JDK v1.8.0_66 +1 compile 0m 44s trunk passed with JDK v1.7.0_85 +1 checkstyle 0m 16s trunk passed +1 mvnsite 0m 52s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 59s trunk passed +1 javadoc 1m 10s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 52s trunk passed with JDK v1.7.0_85 +1 mvninstall 0m 51s the patch passed +1 compile 0m 43s the patch passed with JDK v1.8.0_66 +1 javac 0m 43s the patch passed +1 compile 0m 43s the patch passed with JDK v1.7.0_85 +1 javac 0m 43s the patch passed -1 checkstyle 0m 16s Patch generated 1 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 66, now 67). +1 mvnsite 0m 55s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 11s the patch passed +1 javadoc 1m 4s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 55s the patch passed with JDK v1.7.0_85 -1 unit 52m 58s hadoop-hdfs in the patch failed with JDK v1.8.0_66. +1 unit 49m 29s hadoop-hdfs in the patch passed with JDK v1.7.0_85. -1 asflicense 0m 20s Patch generated 58 ASF License warnings. 130m 8s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes   hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.shortcircuit.TestShortCircuitCache Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12774450/HDFS-9470.002.patch JIRA Issue HDFS-9470 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux f25d2cde9159 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0348e76 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13672/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13672/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13672/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13672/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13672/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 76MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13672/console This message was automatically generated.
        Hide
        xiaochen Xiao Chen added a comment -

        Patch 3 to fix the style warning.
        Failed tests are not related.

        Show
        xiaochen Xiao Chen added a comment - Patch 3 to fix the style warning. Failed tests are not related.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 11m 8s trunk passed
        +1 compile 1m 42s trunk passed with JDK v1.8.0_66
        +1 compile 1m 6s trunk passed with JDK v1.7.0_85
        +1 checkstyle 0m 24s trunk passed
        +1 mvnsite 1m 19s trunk passed
        +1 mvneclipse 0m 18s trunk passed
        +1 findbugs 3m 34s trunk passed
        +1 javadoc 2m 40s trunk passed with JDK v1.8.0_66
        +1 javadoc 3m 24s trunk passed with JDK v1.7.0_85
        +1 mvninstall 1m 32s the patch passed
        +1 compile 1m 51s the patch passed with JDK v1.8.0_66
        +1 javac 1m 51s the patch passed
        +1 compile 1m 32s the patch passed with JDK v1.7.0_85
        +1 javac 1m 32s the patch passed
        +1 checkstyle 0m 29s the patch passed
        +1 mvnsite 1m 51s the patch passed
        +1 mvneclipse 0m 27s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 3m 57s the patch passed
        +1 javadoc 1m 51s the patch passed with JDK v1.8.0_66
        +1 javadoc 2m 52s the patch passed with JDK v1.7.0_85
        -1 unit 117m 1s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
        -1 unit 107m 21s hadoop-hdfs in the patch failed with JDK v1.7.0_85.
        -1 asflicense 0m 31s Patch generated 56 ASF License warnings.
        271m 8s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestBlockStoragePolicy
          hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate
          hadoop.hdfs.server.namenode.ha.TestEditLogTailer
          hadoop.hdfs.TestPersistBlocks
          hadoop.hdfs.security.TestDelegationTokenForProxyUser
          hadoop.hdfs.TestReadStripedFileWithDecoding
          hadoop.hdfs.TestSafeModeWithStripedFile
          hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
          hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation
          hadoop.fs.TestSymlinkHdfsFileContext
          hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA
          hadoop.hdfs.qjournal.TestSecureNNWithQJM
          hadoop.hdfs.server.namenode.TestRecoverStripedBlocks
          hadoop.hdfs.server.datanode.TestDirectoryScanner
        JDK v1.7.0_85 Failed junit tests hadoop.hdfs.TestDFSUpgradeFromImage
          hadoop.hdfs.server.datanode.TestBlockScanner
          hadoop.hdfs.security.TestDelegationTokenForProxyUser
          hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
          hadoop.hdfs.server.datanode.TestBlockReplacement
          hadoop.fs.TestSymlinkHdfsFileContext
          hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA
          hadoop.hdfs.TestAclsEndToEnd
          hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot
          hadoop.hdfs.server.datanode.TestDirectoryScanner



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12774475/HDFS-9470.003.patch
        JIRA Issue HDFS-9470
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 0dd7382c434b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 9f256d1
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13677/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13677/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13677/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13677/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt
        JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13677/testReport/
        asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13677/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Max memory used 75MB
        Powered by Apache Yetus http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13677/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 11m 8s trunk passed +1 compile 1m 42s trunk passed with JDK v1.8.0_66 +1 compile 1m 6s trunk passed with JDK v1.7.0_85 +1 checkstyle 0m 24s trunk passed +1 mvnsite 1m 19s trunk passed +1 mvneclipse 0m 18s trunk passed +1 findbugs 3m 34s trunk passed +1 javadoc 2m 40s trunk passed with JDK v1.8.0_66 +1 javadoc 3m 24s trunk passed with JDK v1.7.0_85 +1 mvninstall 1m 32s the patch passed +1 compile 1m 51s the patch passed with JDK v1.8.0_66 +1 javac 1m 51s the patch passed +1 compile 1m 32s the patch passed with JDK v1.7.0_85 +1 javac 1m 32s the patch passed +1 checkstyle 0m 29s the patch passed +1 mvnsite 1m 51s the patch passed +1 mvneclipse 0m 27s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 3m 57s the patch passed +1 javadoc 1m 51s the patch passed with JDK v1.8.0_66 +1 javadoc 2m 52s the patch passed with JDK v1.7.0_85 -1 unit 117m 1s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 107m 21s hadoop-hdfs in the patch failed with JDK v1.7.0_85. -1 asflicense 0m 31s Patch generated 56 ASF License warnings. 271m 8s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestBlockStoragePolicy   hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate   hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.TestPersistBlocks   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.TestReadStripedFileWithDecoding   hadoop.hdfs.TestSafeModeWithStripedFile   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation   hadoop.fs.TestSymlinkHdfsFileContext   hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA   hadoop.hdfs.qjournal.TestSecureNNWithQJM   hadoop.hdfs.server.namenode.TestRecoverStripedBlocks   hadoop.hdfs.server.datanode.TestDirectoryScanner JDK v1.7.0_85 Failed junit tests hadoop.hdfs.TestDFSUpgradeFromImage   hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.server.datanode.TestBlockReplacement   hadoop.fs.TestSymlinkHdfsFileContext   hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA   hadoop.hdfs.TestAclsEndToEnd   hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot   hadoop.hdfs.server.datanode.TestDirectoryScanner Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12774475/HDFS-9470.003.patch JIRA Issue HDFS-9470 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 0dd7382c434b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 9f256d1 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/13677/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13677/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13677/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13677/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13677/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13677/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 75MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13677/console This message was automatically generated.
        Hide
        djp Junping Du added a comment -

        Hi, can we move this out of 2.6.3? Thanks!

        Show
        djp Junping Du added a comment - Hi, can we move this out of 2.6.3? Thanks!
        Hide
        xiaochen Xiao Chen added a comment -

        Hi Junping Du, I was hoping to make this into 2.6.3 because it's critical, in that after the EZ info on root is lost that's no work around. However due to thanksgiving I'm not sure if this can go in promptly. Ping Andrew Wang for advice. Also please feel free to review this change to accelerate the process. Thanks!

        Show
        xiaochen Xiao Chen added a comment - Hi Junping Du , I was hoping to make this into 2.6.3 because it's critical, in that after the EZ info on root is lost that's no work around. However due to thanksgiving I'm not sure if this can go in promptly. Ping Andrew Wang for advice. Also please feel free to review this change to accelerate the process. Thanks!
        Hide
        wheat9 Haohui Mai added a comment -

        Looks like it is an important fix but it has relatively low risks. I think it is beneficial to put it into 2.6.3.

        The patch looks good to me overall. Kicking off another round of Jenkins run.

        Show
        wheat9 Haohui Mai added a comment - Looks like it is an important fix but it has relatively low risks. I think it is beneficial to put it into 2.6.3. The patch looks good to me overall. Kicking off another round of Jenkins run.
        Hide
        xiaochen Xiao Chen added a comment -

        Thanks Haohui Mai for the comment. I'll watch out for the Jenkins result.

        Show
        xiaochen Xiao Chen added a comment - Thanks Haohui Mai for the comment. I'll watch out for the Jenkins result.
        Hide
        sjlee0 Sangjin Lee added a comment -

        +1 SGTM

        Show
        sjlee0 Sangjin Lee added a comment - +1 SGTM
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 9m 36s trunk passed
        +1 compile 1m 0s trunk passed with JDK v1.8.0_66
        +1 compile 0m 52s trunk passed with JDK v1.7.0_85
        +1 checkstyle 0m 20s trunk passed
        +1 mvnsite 1m 7s trunk passed
        +1 mvneclipse 0m 17s trunk passed
        +1 findbugs 2m 21s trunk passed
        +1 javadoc 1m 27s trunk passed with JDK v1.8.0_66
        +1 javadoc 2m 8s trunk passed with JDK v1.7.0_85
        +1 mvninstall 1m 3s the patch passed
        +1 compile 0m 58s the patch passed with JDK v1.8.0_66
        +1 javac 0m 58s the patch passed
        +1 compile 0m 53s the patch passed with JDK v1.7.0_85
        +1 javac 0m 53s the patch passed
        +1 checkstyle 0m 19s the patch passed
        +1 mvnsite 1m 7s the patch passed
        +1 mvneclipse 0m 16s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 2m 32s the patch passed
        +1 javadoc 1m 24s the patch passed with JDK v1.8.0_66
        +1 javadoc 2m 15s the patch passed with JDK v1.7.0_85
        -1 unit 77m 49s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
        -1 unit 68m 46s hadoop-hdfs in the patch failed with JDK v1.7.0_85.
        -1 asflicense 0m 22s Patch generated 58 ASF License warnings.
        180m 24s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
          hadoop.hdfs.server.namenode.TestDecommissioningStatus
          hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength
          hadoop.hdfs.server.datanode.TestDataNodeMetrics
          hadoop.hdfs.server.datanode.TestBlockScanner
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120
        JDK v1.7.0_85 Failed junit tests hadoop.hdfs.TestEncryptionZones
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure000
          hadoop.hdfs.TestDFSClientRetries
          hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12774475/HDFS-9470.003.patch
        JIRA Issue HDFS-9470
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux a71b00e5fafe 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 43acf9a
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13686/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13686/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13686/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13686/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt
        JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13686/testReport/
        asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13686/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Max memory used 76MB
        Powered by Apache Yetus http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13686/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 9m 36s trunk passed +1 compile 1m 0s trunk passed with JDK v1.8.0_66 +1 compile 0m 52s trunk passed with JDK v1.7.0_85 +1 checkstyle 0m 20s trunk passed +1 mvnsite 1m 7s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 2m 21s trunk passed +1 javadoc 1m 27s trunk passed with JDK v1.8.0_66 +1 javadoc 2m 8s trunk passed with JDK v1.7.0_85 +1 mvninstall 1m 3s the patch passed +1 compile 0m 58s the patch passed with JDK v1.8.0_66 +1 javac 0m 58s the patch passed +1 compile 0m 53s the patch passed with JDK v1.7.0_85 +1 javac 0m 53s the patch passed +1 checkstyle 0m 19s the patch passed +1 mvnsite 1m 7s the patch passed +1 mvneclipse 0m 16s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 32s the patch passed +1 javadoc 1m 24s the patch passed with JDK v1.8.0_66 +1 javadoc 2m 15s the patch passed with JDK v1.7.0_85 -1 unit 77m 49s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 68m 46s hadoop-hdfs in the patch failed with JDK v1.7.0_85. -1 asflicense 0m 22s Patch generated 58 ASF License warnings. 180m 24s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.hdfs.server.namenode.TestDecommissioningStatus   hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength   hadoop.hdfs.server.datanode.TestDataNodeMetrics   hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120 JDK v1.7.0_85 Failed junit tests hadoop.hdfs.TestEncryptionZones   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure000   hadoop.hdfs.TestDFSClientRetries   hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12774475/HDFS-9470.003.patch JIRA Issue HDFS-9470 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux a71b00e5fafe 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 43acf9a findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/13686/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13686/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13686/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13686/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13686/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13686/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 76MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13686/console This message was automatically generated.
        Hide
        xiaochen Xiao Chen added a comment -

        The test failures looks unrelated, and passed locally.

        Show
        xiaochen Xiao Chen added a comment - The test failures looks unrelated, and passed locally.
        Hide
        andrew.wang Andrew Wang added a comment -

        LGTM, I'll commit this down to trunk, branch-2, branch-2.7, branch-2.6.

        Show
        andrew.wang Andrew Wang added a comment - LGTM, I'll commit this down to trunk, branch-2, branch-2.7, branch-2.6.
        Hide
        andrew.wang Andrew Wang added a comment -

        Committed to aforementioned branches, thanks again Xiao Chen for the patch, and all the input from other reviewers.

        Xiao Chen it'd also be nice to get your +1 on the 2.7 and 2.6 cherry-picks, I had to do a small fixup since we don't have the xattr compression patches in those release lines. I compiled and ran TestEncryptionZones for both as a smoke.

        Show
        andrew.wang Andrew Wang added a comment - Committed to aforementioned branches, thanks again Xiao Chen for the patch, and all the input from other reviewers. Xiao Chen it'd also be nice to get your +1 on the 2.7 and 2.6 cherry-picks, I had to do a small fixup since we don't have the xattr compression patches in those release lines. I compiled and ran TestEncryptionZones for both as a smoke.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #8901 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8901/)
        HDFS-9470. Encryption zone on root not loaded from fsimage after NN (wang: rev 9b8e50b424d060e16c1175b1811e7abc476e2468)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZones.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8901 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8901/ ) HDFS-9470 . Encryption zone on root not loaded from fsimage after NN (wang: rev 9b8e50b424d060e16c1175b1811e7abc476e2468) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZones.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
        Hide
        xiaochen Xiao Chen added a comment -

        Thanks Andrew Wang for committing and resolving cherry-pick conflicts. I reviewed the commits to 2.6 and 2.7 branches, LGTM +1.
        Also thanks to everyone for the review and comments.

        Show
        xiaochen Xiao Chen added a comment - Thanks Andrew Wang for committing and resolving cherry-pick conflicts. I reviewed the commits to 2.6 and 2.7 branches, LGTM +1. Also thanks to everyone for the review and comments.
        Hide
        ajisakaa Akira Ajisaka added a comment -

        Hi Xiao Chen, do we need to commit this patch to branch-2.8 as well?

        Show
        ajisakaa Akira Ajisaka added a comment - Hi Xiao Chen , do we need to commit this patch to branch-2.8 as well?
        Hide
        andrew.wang Andrew Wang added a comment -

        Yea my bad on forgetting branch-2.8, I just committed it there too. Thanks Akira Ajisaka for the catch!

        Show
        andrew.wang Andrew Wang added a comment - Yea my bad on forgetting branch-2.8, I just committed it there too. Thanks Akira Ajisaka for the catch!
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #652 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/652/)
        HDFS-9470. Encryption zone on root not loaded from fsimage after NN (wang: rev 9b8e50b424d060e16c1175b1811e7abc476e2468)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZones.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #652 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/652/ ) HDFS-9470 . Encryption zone on root not loaded from fsimage after NN (wang: rev 9b8e50b424d060e16c1175b1811e7abc476e2468) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZones.java
        Hide
        xiaochen Xiao Chen added a comment -
        Show
        xiaochen Xiao Chen added a comment - Thanks Akira Ajisaka and Andrew Wang .
        Hide
        djp Junping Du added a comment -

        Hi Vinod Kumar Vavilapalli, this should be cherry-pick to latest 2.7.2 RC too.

        Show
        djp Junping Du added a comment - Hi Vinod Kumar Vavilapalli , this should be cherry-pick to latest 2.7.2 RC too.
        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        This originally never made it to branch-2.7.2 even though the fix version is set so. Tx to Junping Du for catching this.

        I just cherry-picked it for rolling a new RC for 2.7.2. FYI.

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - This originally never made it to branch-2.7.2 even though the fix version is set so. Tx to Junping Du for catching this. I just cherry-picked it for rolling a new RC for 2.7.2. FYI.

          People

          • Assignee:
            xiaochen Xiao Chen
            Reporter:
            xiaochen Xiao Chen
          • Votes:
            0 Vote for this issue
            Watchers:
            12 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development