Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9804

Allow long-running Balancer to login with keytab

    Details

    • Hadoop Flags:
      Reviewed

      Description

      From the discussion of HDFS-9698, it might be nice to allow the balancer to run as a daemon and login from a keytab.

      1. HDFS-9804.01.patch
        14 kB
        Xiao Chen
      2. HDFS-9804.02.patch
        18 kB
        Xiao Chen
      3. HDFS-9804.03.patch
        17 kB
        Xiao Chen
      4. HDFS-9804-branch-2.00.patch
        17 kB
        Zhe Zhang

        Issue Links

          Activity

          Hide
          cnauroth Chris Nauroth added a comment -

          The "run as a daemon" piece is supported by the shell script rewrite in trunk, i.e. hdfs --daemon start balancer. Supporting login from a keytab would require code changes though.

          Show
          cnauroth Chris Nauroth added a comment - The "run as a daemon" piece is supported by the shell script rewrite in trunk, i.e. hdfs --daemon start balancer . Supporting login from a keytab would require code changes though.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks for the comment Chris Nauroth. I didn't know we have hdfs --daemon.
          Just make sure we're on the same page, we would need to pass in the keytab file to the Balancer, and do something similar to SecurityUtil.login, right?

          Show
          xiaochen Xiao Chen added a comment - Thanks for the comment Chris Nauroth . I didn't know we have hdfs --daemon . Just make sure we're on the same page, we would need to pass in the keytab file to the Balancer, and do something similar to SecurityUtil.login , right?
          Hide
          cnauroth Chris Nauroth added a comment -

          Xiao Chen, yes, that makes sense. It would be an optional call to SecurityUtil#login. By default, it would not attempt to do that (for backwards-compatibility).

          Show
          cnauroth Chris Nauroth added a comment - Xiao Chen , yes, that makes sense. It would be an optional call to SecurityUtil#login . By default, it would not attempt to do that (for backwards-compatibility).
          Hide
          xiaochen Xiao Chen added a comment -

          Patch 1 attached allows the Balancer to login using a keytab.

          • New configurations are added to support this feature. This is disabled by default, so backwards-compatible.
          • In the Balancer, SecurityUtil#login is used to login with keytab.
          • Added a test to verify that logging in from a keytab, the balancer can do exactly the same work as without kerberos. I reused the existing testUnknownDatanode for basic functionality testing, and mimicked TestSecureNNWithQJM#init to setup a kerberized minicluster.
          • Added minor information to UGI#hasSufficientTimeElapsed which I found helpful when identifying this issue.

          I also manually tested that this works in a testing cluster - update the config and start the balancer, the balancer stays logged-in.

          Please review and let me know your thoughts. Thanks!

          Show
          xiaochen Xiao Chen added a comment - Patch 1 attached allows the Balancer to login using a keytab. New configurations are added to support this feature. This is disabled by default, so backwards-compatible. In the Balancer, SecurityUtil#login is used to login with keytab. Added a test to verify that logging in from a keytab, the balancer can do exactly the same work as without kerberos. I reused the existing testUnknownDatanode for basic functionality testing, and mimicked TestSecureNNWithQJM#init to setup a kerberized minicluster. Added minor information to UGI#hasSufficientTimeElapsed which I found helpful when identifying this issue. I also manually tested that this works in a testing cluster - update the config and start the balancer, the balancer stays logged-in. Please review and let me know your thoughts. Thanks!
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 10s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 16s Maven dependency ordering for branch
          +1 mvninstall 6m 56s trunk passed
          +1 compile 5m 58s trunk passed with JDK v1.8.0_72
          +1 compile 6m 40s trunk passed with JDK v1.7.0_95
          +1 checkstyle 1m 11s trunk passed
          +1 mvnsite 1m 56s trunk passed
          +1 mvneclipse 0m 28s trunk passed
          +1 findbugs 3m 31s trunk passed
          +1 javadoc 1m 59s trunk passed with JDK v1.8.0_72
          +1 javadoc 2m 50s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 1m 27s the patch passed
          +1 compile 5m 46s the patch passed with JDK v1.8.0_72
          +1 javac 5m 46s the patch passed
          +1 compile 6m 37s the patch passed with JDK v1.7.0_95
          +1 javac 6m 37s the patch passed
          -1 checkstyle 1m 10s root: patch generated 3 new + 526 unchanged - 0 fixed = 529 total (was 526)
          +1 mvnsite 1m 52s the patch passed
          +1 mvneclipse 0m 27s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 3m 54s the patch passed
          +1 javadoc 1m 55s the patch passed with JDK v1.8.0_72
          +1 javadoc 2m 49s the patch passed with JDK v1.7.0_95
          -1 unit 6m 32s hadoop-common in the patch failed with JDK v1.8.0_72.
          -1 unit 54m 6s hadoop-hdfs in the patch failed with JDK v1.8.0_72.
          +1 unit 7m 13s hadoop-common in the patch passed with JDK v1.7.0_95.
          -1 unit 52m 18s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 25s Patch does not generate ASF License warnings.
          180m 0s



          Reason Tests
          JDK v1.8.0_72 Failed junit tests hadoop.ha.TestZKFailoverController
            hadoop.hdfs.server.balancer.TestBalancer
            hadoop.hdfs.TestFileAppend
          JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.balancer.TestBalancer



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12788320/HDFS-9804.01.patch
          JIRA Issue HDFS-9804
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 4544a6899122 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 0fb14aa
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14516/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14516/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_72.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14516/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14516/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14516/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14516/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14516/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14516/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14516/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 16s Maven dependency ordering for branch +1 mvninstall 6m 56s trunk passed +1 compile 5m 58s trunk passed with JDK v1.8.0_72 +1 compile 6m 40s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 11s trunk passed +1 mvnsite 1m 56s trunk passed +1 mvneclipse 0m 28s trunk passed +1 findbugs 3m 31s trunk passed +1 javadoc 1m 59s trunk passed with JDK v1.8.0_72 +1 javadoc 2m 50s trunk passed with JDK v1.7.0_95 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 27s the patch passed +1 compile 5m 46s the patch passed with JDK v1.8.0_72 +1 javac 5m 46s the patch passed +1 compile 6m 37s the patch passed with JDK v1.7.0_95 +1 javac 6m 37s the patch passed -1 checkstyle 1m 10s root: patch generated 3 new + 526 unchanged - 0 fixed = 529 total (was 526) +1 mvnsite 1m 52s the patch passed +1 mvneclipse 0m 27s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 3m 54s the patch passed +1 javadoc 1m 55s the patch passed with JDK v1.8.0_72 +1 javadoc 2m 49s the patch passed with JDK v1.7.0_95 -1 unit 6m 32s hadoop-common in the patch failed with JDK v1.8.0_72. -1 unit 54m 6s hadoop-hdfs in the patch failed with JDK v1.8.0_72. +1 unit 7m 13s hadoop-common in the patch passed with JDK v1.7.0_95. -1 unit 52m 18s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 25s Patch does not generate ASF License warnings. 180m 0s Reason Tests JDK v1.8.0_72 Failed junit tests hadoop.ha.TestZKFailoverController   hadoop.hdfs.server.balancer.TestBalancer   hadoop.hdfs.TestFileAppend JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.balancer.TestBalancer Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12788320/HDFS-9804.01.patch JIRA Issue HDFS-9804 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 4544a6899122 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0fb14aa Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14516/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14516/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14516/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14516/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14516/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14516/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14516/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14516/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14516/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Failed test is related, and I just realized I should add the configs to hdfs-default.xml. Will post a patch soon.

          Show
          xiaochen Xiao Chen added a comment - Failed test is related, and I just realized I should add the configs to hdfs-default.xml. Will post a patch soon.
          Hide
          xiaochen Xiao Chen added a comment -

          Patch 2 adds the configs into hdfs-default.xml, and fixes the unit tests (passed locally).

          Show
          xiaochen Xiao Chen added a comment - Patch 2 adds the configs into hdfs-default.xml, and fixes the unit tests (passed locally).
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 4s HDFS-9804 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12788485/HDFS-9804.02.patch
          JIRA Issue HDFS-9804
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14530/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 4s HDFS-9804 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12788485/HDFS-9804.02.patch JIRA Issue HDFS-9804 Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14530/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Oops, hdfs-default.xml was just changed recently. Attached an updated patch 2.

          Show
          xiaochen Xiao Chen added a comment - Oops, hdfs-default.xml was just changed recently. Attached an updated patch 2.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 15s Maven dependency ordering for branch
          +1 mvninstall 6m 33s trunk passed
          +1 compile 5m 53s trunk passed with JDK v1.8.0_72
          +1 compile 6m 48s trunk passed with JDK v1.7.0_95
          +1 checkstyle 1m 8s trunk passed
          +1 mvnsite 1m 54s trunk passed
          +1 mvneclipse 0m 27s trunk passed
          +1 findbugs 3m 27s trunk passed
          +1 javadoc 2m 1s trunk passed with JDK v1.8.0_72
          +1 javadoc 2m 47s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 1m 26s the patch passed
          +1 compile 5m 55s the patch passed with JDK v1.8.0_72
          +1 javac 5m 55s the patch passed
          +1 compile 6m 39s the patch passed with JDK v1.7.0_95
          +1 javac 6m 40s the patch passed
          -1 checkstyle 1m 9s root: patch generated 3 new + 528 unchanged - 0 fixed = 531 total (was 528)
          +1 mvnsite 1m 52s the patch passed
          +1 mvneclipse 0m 27s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 3m 57s the patch passed
          +1 javadoc 2m 1s the patch passed with JDK v1.8.0_72
          +1 javadoc 2m 50s the patch passed with JDK v1.7.0_95
          +1 unit 7m 22s hadoop-common in the patch passed with JDK v1.8.0_72.
          -1 unit 55m 19s hadoop-hdfs in the patch failed with JDK v1.8.0_72.
          +1 unit 7m 17s hadoop-common in the patch passed with JDK v1.7.0_95.
          +1 unit 51m 24s hadoop-hdfs in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 25s Patch does not generate ASF License warnings.
          181m 5s



          Reason Tests
          JDK v1.8.0_72 Failed junit tests hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork
            hadoop.hdfs.server.datanode.TestDataNodeMetrics
            hadoop.hdfs.TestCrcCorruption
            hadoop.hdfs.shortcircuit.TestShortCircuitCache



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12788495/HDFS-9804.02.patch
          JIRA Issue HDFS-9804
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 0b90329d4853 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / ba6d5ed
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14531/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14531/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14531/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14531/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14531/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 15s Maven dependency ordering for branch +1 mvninstall 6m 33s trunk passed +1 compile 5m 53s trunk passed with JDK v1.8.0_72 +1 compile 6m 48s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 8s trunk passed +1 mvnsite 1m 54s trunk passed +1 mvneclipse 0m 27s trunk passed +1 findbugs 3m 27s trunk passed +1 javadoc 2m 1s trunk passed with JDK v1.8.0_72 +1 javadoc 2m 47s trunk passed with JDK v1.7.0_95 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 26s the patch passed +1 compile 5m 55s the patch passed with JDK v1.8.0_72 +1 javac 5m 55s the patch passed +1 compile 6m 39s the patch passed with JDK v1.7.0_95 +1 javac 6m 40s the patch passed -1 checkstyle 1m 9s root: patch generated 3 new + 528 unchanged - 0 fixed = 531 total (was 528) +1 mvnsite 1m 52s the patch passed +1 mvneclipse 0m 27s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 3m 57s the patch passed +1 javadoc 2m 1s the patch passed with JDK v1.8.0_72 +1 javadoc 2m 50s the patch passed with JDK v1.7.0_95 +1 unit 7m 22s hadoop-common in the patch passed with JDK v1.8.0_72. -1 unit 55m 19s hadoop-hdfs in the patch failed with JDK v1.8.0_72. +1 unit 7m 17s hadoop-common in the patch passed with JDK v1.7.0_95. +1 unit 51m 24s hadoop-hdfs in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 25s Patch does not generate ASF License warnings. 181m 5s Reason Tests JDK v1.8.0_72 Failed junit tests hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork   hadoop.hdfs.server.datanode.TestDataNodeMetrics   hadoop.hdfs.TestCrcCorruption   hadoop.hdfs.shortcircuit.TestShortCircuitCache Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12788495/HDFS-9804.02.patch JIRA Issue HDFS-9804 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 0b90329d4853 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ba6d5ed Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14531/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14531/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14531/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14531/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14531/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Test failures seems unrelated. I'm gonna ignore the checkstyles since that's all the config related definitions.

          Show
          xiaochen Xiao Chen added a comment - Test failures seems unrelated. I'm gonna ignore the checkstyles since that's all the config related definitions.
          Hide
          zhz Zhe Zhang added a comment -

          Thanks Xiao for the fix. The patch looks to me almost ready to go. A few minors:

          1. It's better to add some error handling in checkKerberosAndInit: what if DFS_BALANCER_KERBEROS_PRINCIPAL_KEY or the keytab file key is not set.
          2. Looks like getAddress can be folded into the checkKerberosAndInit method?
          3. Maybe checkKeytabAndInit is a better name?
          4. assertTrue(ugi.isLoginKeytabBased()) should be UserGroupInformation.isLoginKeytabBased() since the method is static
          5. This can be a follow-on: ideally we can verify the behavior when used with the hdfs --daemon option.
          6. Another follow-on idea is to verify the relogin after TGT "max renew time" expires. It could be hard to control KDC TGT config though.
          Show
          zhz Zhe Zhang added a comment - Thanks Xiao for the fix. The patch looks to me almost ready to go. A few minors: It's better to add some error handling in checkKerberosAndInit : what if DFS_BALANCER_KERBEROS_PRINCIPAL_KEY or the keytab file key is not set. Looks like getAddress can be folded into the checkKerberosAndInit method? Maybe checkKeytabAndInit is a better name? assertTrue(ugi.isLoginKeytabBased()) should be UserGroupInformation.isLoginKeytabBased() since the method is static This can be a follow-on: ideally we can verify the behavior when used with the hdfs --daemon option. Another follow-on idea is to verify the relogin after TGT "max renew time" expires. It could be hard to control KDC TGT config though.
          Hide
          zhz Zhe Zhang added a comment -

          Chris Nauroth Pinging to check if you have other comments. Thanks for the initial review!

          Show
          zhz Zhe Zhang added a comment - Chris Nauroth Pinging to check if you have other comments. Thanks for the initial review!
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Zhe Zhang for the review! Patch 3 is attached.

          It's better to add some error handling in checkKerberosAndInit: what if DFS_BALANCER_KERBEROS_PRINCIPAL_KEY or the keytab file key is not set.

          That's handled in SecurityUtil#login
          It'll throw IOE with details if keytab is given incorrectly, and will use system username if principal is not provided

          Looks like getAddress can be folded into the checkKerberosAndInit method?

          Sure.

          Maybe checkKeytabAndInit is a better name?

          Agreed.

          assertTrue(ugi.isLoginKeytabBased()) should be UserGroupInformation.isLoginKeytabBased() since the method is static

          Done

          This can be a follow-on: ideally we can verify the behavior when used with the hdfs --daemon option.

          Makes sense, I assume --daemon would be the same for all commands though.

          Another follow-on idea is to verify the relogin after TGT "max renew time" expires. It could be hard to control KDC TGT config though.

          I manually verified this, but I guess I could borrow your patch from HADOOP-12559 to do it programmatically.

          Show
          xiaochen Xiao Chen added a comment - Thanks Zhe Zhang for the review! Patch 3 is attached. It's better to add some error handling in checkKerberosAndInit: what if DFS_BALANCER_KERBEROS_PRINCIPAL_KEY or the keytab file key is not set. That's handled in SecurityUtil#login It'll throw IOE with details if keytab is given incorrectly, and will use system username if principal is not provided Looks like getAddress can be folded into the checkKerberosAndInit method? Sure. Maybe checkKeytabAndInit is a better name? Agreed. assertTrue(ugi.isLoginKeytabBased()) should be UserGroupInformation.isLoginKeytabBased() since the method is static Done This can be a follow-on: ideally we can verify the behavior when used with the hdfs --daemon option. Makes sense, I assume --daemon would be the same for all commands though. Another follow-on idea is to verify the relogin after TGT "max renew time" expires. It could be hard to control KDC TGT config though. I manually verified this, but I guess I could borrow your patch from HADOOP-12559 to do it programmatically.
          Hide
          zhz Zhe Zhang added a comment -

          Thanks Xiao for the update and clarification! +1 on the v3 patch pending Jenkins.

          Show
          zhz Zhe Zhang added a comment - Thanks Xiao for the update and clarification! +1 on the v3 patch pending Jenkins.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Zhe! I also updated the title of the jira to be more specific.

          Show
          xiaochen Xiao Chen added a comment - Thanks Zhe! I also updated the title of the jira to be more specific.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 12s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 34s Maven dependency ordering for branch
          +1 mvninstall 6m 59s trunk passed
          +1 compile 6m 40s trunk passed with JDK v1.8.0_72
          +1 compile 6m 37s trunk passed with JDK v1.7.0_95
          +1 checkstyle 1m 9s trunk passed
          +1 mvnsite 1m 55s trunk passed
          +1 mvneclipse 0m 27s trunk passed
          +1 findbugs 3m 28s trunk passed
          +1 javadoc 2m 1s trunk passed with JDK v1.8.0_72
          +1 javadoc 2m 48s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 1m 25s the patch passed
          +1 compile 5m 46s the patch passed with JDK v1.8.0_72
          +1 javac 5m 46s the patch passed
          +1 compile 6m 36s the patch passed with JDK v1.7.0_95
          +1 javac 6m 36s the patch passed
          -1 checkstyle 1m 9s root: patch generated 3 new + 527 unchanged - 0 fixed = 530 total (was 527)
          +1 mvnsite 1m 53s the patch passed
          +1 mvneclipse 0m 27s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 3m 55s the patch passed
          +1 javadoc 1m 59s the patch passed with JDK v1.8.0_72
          +1 javadoc 2m 47s the patch passed with JDK v1.7.0_95
          +1 unit 6m 43s hadoop-common in the patch passed with JDK v1.8.0_72.
          -1 unit 53m 36s hadoop-hdfs in the patch failed with JDK v1.8.0_72.
          -1 unit 7m 49s hadoop-common in the patch failed with JDK v1.7.0_95.
          -1 unit 53m 51s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 25s Patch does not generate ASF License warnings.
          182m 49s



          Reason Tests
          JDK v1.8.0_72 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeMetrics
          JDK v1.7.0_95 Failed junit tests hadoop.ha.TestZKFailoverController
            hadoop.hdfs.server.datanode.TestDataNodeMetrics



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12789803/HDFS-9804.03.patch
          JIRA Issue HDFS-9804
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 9104c6f19fc4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / dbbfc58
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14605/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14605/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14605/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14605/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14605/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14605/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14605/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14605/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14605/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 34s Maven dependency ordering for branch +1 mvninstall 6m 59s trunk passed +1 compile 6m 40s trunk passed with JDK v1.8.0_72 +1 compile 6m 37s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 9s trunk passed +1 mvnsite 1m 55s trunk passed +1 mvneclipse 0m 27s trunk passed +1 findbugs 3m 28s trunk passed +1 javadoc 2m 1s trunk passed with JDK v1.8.0_72 +1 javadoc 2m 48s trunk passed with JDK v1.7.0_95 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 25s the patch passed +1 compile 5m 46s the patch passed with JDK v1.8.0_72 +1 javac 5m 46s the patch passed +1 compile 6m 36s the patch passed with JDK v1.7.0_95 +1 javac 6m 36s the patch passed -1 checkstyle 1m 9s root: patch generated 3 new + 527 unchanged - 0 fixed = 530 total (was 527) +1 mvnsite 1m 53s the patch passed +1 mvneclipse 0m 27s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 3m 55s the patch passed +1 javadoc 1m 59s the patch passed with JDK v1.8.0_72 +1 javadoc 2m 47s the patch passed with JDK v1.7.0_95 +1 unit 6m 43s hadoop-common in the patch passed with JDK v1.8.0_72. -1 unit 53m 36s hadoop-hdfs in the patch failed with JDK v1.8.0_72. -1 unit 7m 49s hadoop-common in the patch failed with JDK v1.7.0_95. -1 unit 53m 51s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 25s Patch does not generate ASF License warnings. 182m 49s Reason Tests JDK v1.8.0_72 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeMetrics JDK v1.7.0_95 Failed junit tests hadoop.ha.TestZKFailoverController   hadoop.hdfs.server.datanode.TestDataNodeMetrics Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12789803/HDFS-9804.03.patch JIRA Issue HDFS-9804 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 9104c6f19fc4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / dbbfc58 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14605/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14605/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14605/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14605/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14605/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14605/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14605/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14605/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14605/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          The checkstyle is DFSConfigKeys over 80 chars as before.
          The failed tests, although TestDataNodeMetrics failed on both with the same error, is unrelated and passed locally. There's HDFS-9766 tracking this intermittent test.

          Show
          xiaochen Xiao Chen added a comment - The checkstyle is DFSConfigKeys over 80 chars as before. The failed tests, although TestDataNodeMetrics failed on both with the same error, is unrelated and passed locally. There's HDFS-9766 tracking this intermittent test.
          Hide
          zhz Zhe Zhang added a comment -

          Thanks Xiao. The reported failures are unrelated to Balancer. I just committed the patch to trunk. I don't think the change is very useful without the daemon option? LMK if you think it should go into branch-2 as well.

          Show
          zhz Zhe Zhang added a comment - Thanks Xiao. The reported failures are unrelated to Balancer. I just committed the patch to trunk. I don't think the change is very useful without the daemon option? LMK if you think it should go into branch-2 as well.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9368 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9368/)
          HDFS-9804. Allow long-running Balancer to login with keytab. Contributed (zhz: rev ccff6035f50769eb69701128ae61efc69e82609d)

          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9368 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9368/ ) HDFS-9804 . Allow long-running Balancer to login with keytab. Contributed (zhz: rev ccff6035f50769eb69701128ae61efc69e82609d) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks again Zhe.

          The reported failures are unrelated to Balancer.

          Yep, I posted a patch to HDFS-9766 on that.

          LMK if you think it should go into branch-2 as well.

          I think it's Okay to not go into branch-2. If the cluster is super imbalanced, the balancer process may not finish within the TGT max renew time, but I guess that's a rare case and no one seems to be complaining.

          Show
          xiaochen Xiao Chen added a comment - Thanks again Zhe. The reported failures are unrelated to Balancer. Yep, I posted a patch to HDFS-9766 on that. LMK if you think it should go into branch-2 as well. I think it's Okay to not go into branch-2. If the cluster is super imbalanced, the balancer process may not finish within the TGT max renew time, but I guess that's a rare case and no one seems to be complaining.
          Hide
          zhz Zhe Zhang added a comment -

          Actually the issue is still valid in branch-2. Some large clusters could use multiple days to finish the balancing process. Therefore the admin could configure a keytab file and also skip the kinit step. In that case, Balancer won't start without the patch. I'm attaching a branch-2 patch to trigger Jenkins.

          Show
          zhz Zhe Zhang added a comment - Actually the issue is still valid in branch-2. Some large clusters could use multiple days to finish the balancing process. Therefore the admin could configure a keytab file and also skip the kinit step. In that case, Balancer won't start without the patch. I'm attaching a branch-2 patch to trigger Jenkins.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Zhe Zhang!
          Not very related but just FYI - there are follow-on jiras (linked) to fix the unit test being flaky, and eventually there's HDFS-10716 which may prevent the balancer from finishing.

          Show
          xiaochen Xiao Chen added a comment - Thanks Zhe Zhang ! Not very related but just FYI - there are follow-on jiras (linked) to fix the unit test being flaky, and eventually there's HDFS-10716 which may prevent the balancer from finishing.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 55s Maven dependency ordering for branch
          +1 mvninstall 6m 37s branch-2 passed
          +1 compile 5m 30s branch-2 passed with JDK v1.8.0_101
          +1 compile 6m 24s branch-2 passed with JDK v1.7.0_101
          +1 checkstyle 1m 30s branch-2 passed
          +1 mvnsite 1m 50s branch-2 passed
          +1 mvneclipse 0m 32s branch-2 passed
          +1 findbugs 3m 35s branch-2 passed
          +1 javadoc 1m 41s branch-2 passed with JDK v1.8.0_101
          +1 javadoc 2m 32s branch-2 passed with JDK v1.7.0_101
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 1m 27s the patch passed
          +1 compile 5m 26s the patch passed with JDK v1.8.0_101
          +1 javac 5m 26s the patch passed
          +1 compile 6m 35s the patch passed with JDK v1.7.0_101
          +1 javac 6m 35s the patch passed
          -0 checkstyle 1m 30s root: The patch generated 5 new + 700 unchanged - 0 fixed = 705 total (was 700)
          +1 mvnsite 1m 53s the patch passed
          +1 mvneclipse 0m 31s the patch passed
          -1 whitespace 0m 0s The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          +1 xml 0m 0s The patch has no ill-formed XML file.
          +1 findbugs 4m 5s the patch passed
          +1 javadoc 1m 41s the patch passed with JDK v1.8.0_101
          +1 javadoc 2m 31s the patch passed with JDK v1.7.0_101
          +1 unit 7m 33s hadoop-common in the patch passed with JDK v1.7.0_101.
          -1 unit 50m 10s hadoop-hdfs in the patch failed with JDK v1.7.0_101.
          +1 asflicense 0m 27s The patch does not generate ASF License warnings.
          173m 20s



          Reason Tests
          JDK v1.8.0_101 Failed junit tests hadoop.hdfs.server.balancer.TestBalancer
          JDK v1.7.0_101 Failed junit tests hadoop.hdfs.TestEncryptionZones
            hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency
            hadoop.hdfs.server.balancer.TestBalancer



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:b59b8b7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12823130/HDFS-9804-branch-2.00.patch
          JIRA Issue HDFS-9804
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 69b88ff44efe 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / 8efd495
          Default Java 1.7.0_101
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16394/artifact/patchprocess/diff-checkstyle-root.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/16394/artifact/patchprocess/whitespace-eol.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16394/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_101.txt
          JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16394/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16394/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 55s Maven dependency ordering for branch +1 mvninstall 6m 37s branch-2 passed +1 compile 5m 30s branch-2 passed with JDK v1.8.0_101 +1 compile 6m 24s branch-2 passed with JDK v1.7.0_101 +1 checkstyle 1m 30s branch-2 passed +1 mvnsite 1m 50s branch-2 passed +1 mvneclipse 0m 32s branch-2 passed +1 findbugs 3m 35s branch-2 passed +1 javadoc 1m 41s branch-2 passed with JDK v1.8.0_101 +1 javadoc 2m 32s branch-2 passed with JDK v1.7.0_101 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 27s the patch passed +1 compile 5m 26s the patch passed with JDK v1.8.0_101 +1 javac 5m 26s the patch passed +1 compile 6m 35s the patch passed with JDK v1.7.0_101 +1 javac 6m 35s the patch passed -0 checkstyle 1m 30s root: The patch generated 5 new + 700 unchanged - 0 fixed = 705 total (was 700) +1 mvnsite 1m 53s the patch passed +1 mvneclipse 0m 31s the patch passed -1 whitespace 0m 0s The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 xml 0m 0s The patch has no ill-formed XML file. +1 findbugs 4m 5s the patch passed +1 javadoc 1m 41s the patch passed with JDK v1.8.0_101 +1 javadoc 2m 31s the patch passed with JDK v1.7.0_101 +1 unit 7m 33s hadoop-common in the patch passed with JDK v1.7.0_101. -1 unit 50m 10s hadoop-hdfs in the patch failed with JDK v1.7.0_101. +1 asflicense 0m 27s The patch does not generate ASF License warnings. 173m 20s Reason Tests JDK v1.8.0_101 Failed junit tests hadoop.hdfs.server.balancer.TestBalancer JDK v1.7.0_101 Failed junit tests hadoop.hdfs.TestEncryptionZones   hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency   hadoop.hdfs.server.balancer.TestBalancer Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12823130/HDFS-9804-branch-2.00.patch JIRA Issue HDFS-9804 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 69b88ff44efe 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 8efd495 Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16394/artifact/patchprocess/diff-checkstyle-root.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/16394/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16394/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_101.txt JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16394/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16394/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 21s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 1m 3s Maven dependency ordering for branch
          +1 mvninstall 7m 8s branch-2 passed
          +1 compile 5m 56s branch-2 passed with JDK v1.8.0_101
          +1 compile 6m 50s branch-2 passed with JDK v1.7.0_101
          +1 checkstyle 1m 46s branch-2 passed
          +1 mvnsite 1m 58s branch-2 passed
          +1 mvneclipse 0m 32s branch-2 passed
          +1 findbugs 3m 46s branch-2 passed
          +1 javadoc 1m 46s branch-2 passed with JDK v1.8.0_101
          +1 javadoc 2m 37s branch-2 passed with JDK v1.7.0_101
          0 mvndep 0m 13s Maven dependency ordering for patch
          +1 mvninstall 1m 32s the patch passed
          +1 compile 6m 2s the patch passed with JDK v1.8.0_101
          +1 javac 6m 2s the patch passed
          +1 compile 7m 1s the patch passed with JDK v1.7.0_101
          +1 javac 7m 1s the patch passed
          -0 checkstyle 1m 39s root: The patch generated 5 new + 701 unchanged - 0 fixed = 706 total (was 701)
          +1 mvnsite 2m 6s the patch passed
          +1 mvneclipse 0m 38s the patch passed
          -1 whitespace 0m 0s The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 4m 28s the patch passed
          +1 javadoc 1m 53s the patch passed with JDK v1.8.0_101
          +1 javadoc 2m 53s the patch passed with JDK v1.7.0_101
          +1 unit 9m 22s hadoop-common in the patch passed with JDK v1.7.0_101.
          -1 unit 62m 6s hadoop-hdfs in the patch failed with JDK v1.7.0_101.
          +1 asflicense 0m 24s The patch does not generate ASF License warnings.
          212m 6s



          Reason Tests
          JDK v1.8.0_101 Failed junit tests hadoop.hdfs.server.balancer.TestBalancer
          JDK v1.7.0_101 Failed junit tests hadoop.hdfs.server.balancer.TestBalancer



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:b59b8b7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12823130/HDFS-9804-branch-2.00.patch
          JIRA Issue HDFS-9804
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 1071f567a54b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / 8efd495
          Default Java 1.7.0_101
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16395/artifact/patchprocess/diff-checkstyle-root.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/16395/artifact/patchprocess/whitespace-eol.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16395/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_101.txt
          JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16395/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16395/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 1m 3s Maven dependency ordering for branch +1 mvninstall 7m 8s branch-2 passed +1 compile 5m 56s branch-2 passed with JDK v1.8.0_101 +1 compile 6m 50s branch-2 passed with JDK v1.7.0_101 +1 checkstyle 1m 46s branch-2 passed +1 mvnsite 1m 58s branch-2 passed +1 mvneclipse 0m 32s branch-2 passed +1 findbugs 3m 46s branch-2 passed +1 javadoc 1m 46s branch-2 passed with JDK v1.8.0_101 +1 javadoc 2m 37s branch-2 passed with JDK v1.7.0_101 0 mvndep 0m 13s Maven dependency ordering for patch +1 mvninstall 1m 32s the patch passed +1 compile 6m 2s the patch passed with JDK v1.8.0_101 +1 javac 6m 2s the patch passed +1 compile 7m 1s the patch passed with JDK v1.7.0_101 +1 javac 7m 1s the patch passed -0 checkstyle 1m 39s root: The patch generated 5 new + 701 unchanged - 0 fixed = 706 total (was 701) +1 mvnsite 2m 6s the patch passed +1 mvneclipse 0m 38s the patch passed -1 whitespace 0m 0s The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 4m 28s the patch passed +1 javadoc 1m 53s the patch passed with JDK v1.8.0_101 +1 javadoc 2m 53s the patch passed with JDK v1.7.0_101 +1 unit 9m 22s hadoop-common in the patch passed with JDK v1.7.0_101. -1 unit 62m 6s hadoop-hdfs in the patch failed with JDK v1.7.0_101. +1 asflicense 0m 24s The patch does not generate ASF License warnings. 212m 6s Reason Tests JDK v1.8.0_101 Failed junit tests hadoop.hdfs.server.balancer.TestBalancer JDK v1.7.0_101 Failed junit tests hadoop.hdfs.server.balancer.TestBalancer Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12823130/HDFS-9804-branch-2.00.patch JIRA Issue HDFS-9804 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 1071f567a54b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 8efd495 Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16395/artifact/patchprocess/diff-checkstyle-root.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/16395/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16395/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_101.txt JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16395/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16395/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          zhz Zhe Zhang added a comment -

          I just verified the backports to branch-2 and branch-2.8, and committed to the branches. Thanks Xiao for the info. I backported the 2 test-related patches as well: HDFS-10336 and HDFS-9888.

          Show
          zhz Zhe Zhang added a comment - I just verified the backports to branch-2 and branch-2.8, and committed to the branches. Thanks Xiao for the info. I backported the 2 test-related patches as well: HDFS-10336 and HDFS-9888 .

            People

            • Assignee:
              xiaochen Xiao Chen
              Reporter:
              xiaochen Xiao Chen
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development