Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9888

Allow reseting KerberosName in unit tests

    Details

    • Type: Test
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 2.7.4, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      In some local environments, TestBalancer#testBalancerWithKeytabs may fail. Specifically, running itself passes, but running TestBalancer suite always fail. This is due to:

      1. Kerberos setup is done at the test case setup
      2. static variable KerberosName#defaultRealm is set when class initialization - before testBalancerWithKeytabs setup
      3. local default realm is different than test case default realm

      This is mostly an environment specific problem, but let's not make such assumption in the test.

        Issue Links

          Activity

          Hide
          xiaochen Xiao Chen added a comment -

          Step 2 in the description is here

            static {
              try {
                defaultRealm = KerberosUtil.getDefaultRealm();
              } catch (Exception ke) {
                  LOG.debug("Kerberos krb5 configuration not found, setting default realm to empty");
                  defaultRealm="";
              }
            }
          

          So patch 1 adds a test-only method to reset the static var defaultRealm. Also broadened the try-finally block in TestBalancer#testBalancerWithKeytabs - currently the try is late.

          Zhe Zhang, could you help review? I've verified the patch in the local env in a fail-before, pass-after manner. Thanks!

          Show
          xiaochen Xiao Chen added a comment - Step 2 in the description is here static { try { defaultRealm = KerberosUtil.getDefaultRealm(); } catch (Exception ke) { LOG.debug( "Kerberos krb5 configuration not found, setting default realm to empty" ); defaultRealm=""; } } So patch 1 adds a test-only method to reset the static var defaultRealm . Also broadened the try-finally block in TestBalancer#testBalancerWithKeytabs - currently the try is late. Zhe Zhang , could you help review? I've verified the patch in the local env in a fail-before, pass-after manner. Thanks!
          Hide
          zhz Zhe Zhang added a comment -

          Thanks Xiao, idea sounds good. Patch cannot apply though.

          Show
          zhz Zhe Zhang added a comment - Thanks Xiao, idea sounds good. Patch cannot apply though.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Zhe for the quick response. I just double checked with latest trunk, applied for me.... Could you share which part was the conflict?

          Show
          xiaochen Xiao Chen added a comment - Thanks Zhe for the quick response. I just double checked with latest trunk, applied for me.... Could you share which part was the conflict?
          Hide
          zhz Zhe Zhang added a comment -

          My bad, tried wrong patch. Reviewing now.

          Show
          zhz Zhe Zhang added a comment - My bad, tried wrong patch. Reviewing now.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks!

          Show
          xiaochen Xiao Chen added a comment - Thanks!
          Hide
          zhz Zhe Zhang added a comment -

          Thanks Xiao. Could you paste the stacktrace of failed test runs?

          Show
          zhz Zhe Zhang added a comment - Thanks Xiao. Could you paste the stacktrace of failed test runs?
          Hide
          xiaochen Xiao Chen added a comment -

          Sure Zhe, here's a sample error:

          Error Message
          
          Login failure for balancer/localhost@EXAMPLE.COM from keytab /data/jenkins/workspace/CDH5-Hadoop-HDFS-2.6.0/hadoop-hdfs-project/hadoop-hdfs/target/test-dir/TestBalancer/balancer.keytab
          Stacktrace
          
          java.io.IOException: Login failure for balancer/localhost@EXAMPLE.COM from keytab /data/jenkins/workspace/CDH5-Hadoop-HDFS-2.6.0/hadoop-hdfs-project/hadoop-hdfs/target/test-dir/TestBalancer/balancer.keytab
          	at org.apache.hadoop.security.authentication.util.KerberosName.getShortName(KerberosName.java:389)
          	at org.apache.hadoop.security.User.<init>(User.java:48)
          	at org.apache.hadoop.security.User.<init>(User.java:43)
          	at org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.commit(UserGroupInformation.java:197)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at java.lang.reflect.Method.invoke(Method.java:606)
          	at javax.security.auth.login.LoginContext.invoke(LoginContext.java:762)
          	at javax.security.auth.login.LoginContext.access$000(LoginContext.java:203)
          	at javax.security.auth.login.LoginContext$4.run(LoginContext.java:690)
          	at javax.security.auth.login.LoginContext$4.run(LoginContext.java:688)
          	at java.security.AccessController.doPrivileged(Native Method)
          	at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:687)
          	at javax.security.auth.login.LoginContext.login(LoginContext.java:596)
          	at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1171)
          	at org.apache.hadoop.hdfs.server.balancer.TestBalancer.testBalancerWithKeytabs(TestBalancer.java:1594)
          

          Basically the exception above is from KerberosName cannot find rules to apply. Further logging shows that the realm locally is 'APACHE.ORG' instead of 'EXAMPLE.COM'. (a change by HDFS-3016 back in 2012, which is not present locally. Even if that's present, this patch will make the test safer.)

          Show
          xiaochen Xiao Chen added a comment - Sure Zhe, here's a sample error: Error Message Login failure for balancer/localhost@EXAMPLE.COM from keytab /data/jenkins/workspace/CDH5-Hadoop-HDFS-2.6.0/hadoop-hdfs-project/hadoop-hdfs/target/test-dir/TestBalancer/balancer.keytab Stacktrace java.io.IOException: Login failure for balancer/localhost@EXAMPLE.COM from keytab /data/jenkins/workspace/CDH5-Hadoop-HDFS-2.6.0/hadoop-hdfs-project/hadoop-hdfs/target/test-dir/TestBalancer/balancer.keytab at org.apache.hadoop.security.authentication.util.KerberosName.getShortName(KerberosName.java:389) at org.apache.hadoop.security.User.<init>(User.java:48) at org.apache.hadoop.security.User.<init>(User.java:43) at org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.commit(UserGroupInformation.java:197) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at javax.security.auth.login.LoginContext.invoke(LoginContext.java:762) at javax.security.auth.login.LoginContext.access$000(LoginContext.java:203) at javax.security.auth.login.LoginContext$4.run(LoginContext.java:690) at javax.security.auth.login.LoginContext$4.run(LoginContext.java:688) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:687) at javax.security.auth.login.LoginContext.login(LoginContext.java:596) at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1171) at org.apache.hadoop.hdfs.server.balancer.TestBalancer.testBalancerWithKeytabs(TestBalancer.java:1594) Basically the exception above is from KerberosName cannot find rules to apply. Further logging shows that the realm locally is 'APACHE.ORG' instead of 'EXAMPLE.COM'. (a change by HDFS-3016 back in 2012, which is not present locally. Even if that's present, this patch will make the test safer.)
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 12s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 37s Maven dependency ordering for branch
          +1 mvninstall 8m 16s trunk passed
          +1 compile 11m 13s trunk passed with JDK v1.8.0_72
          +1 compile 8m 30s trunk passed with JDK v1.7.0_95
          +1 checkstyle 1m 8s trunk passed
          +1 mvnsite 1m 25s trunk passed
          +1 mvneclipse 0m 27s trunk passed
          +1 findbugs 2m 37s trunk passed
          +1 javadoc 1m 40s trunk passed with JDK v1.8.0_72
          +1 javadoc 2m 21s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvninstall 1m 13s the patch passed
          +1 compile 10m 26s the patch passed with JDK v1.8.0_72
          +1 javac 10m 26s the patch passed
          +1 compile 8m 55s the patch passed with JDK v1.7.0_95
          +1 javac 8m 55s the patch passed
          +1 checkstyle 1m 9s the patch passed
          +1 mvnsite 1m 25s the patch passed
          +1 mvneclipse 0m 30s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 3m 5s the patch passed
          +1 javadoc 1m 42s the patch passed with JDK v1.8.0_72
          +1 javadoc 2m 27s the patch passed with JDK v1.7.0_95
          +1 unit 3m 58s hadoop-auth in the patch passed with JDK v1.8.0_72.
          -1 unit 74m 40s hadoop-hdfs in the patch failed with JDK v1.8.0_72.
          +1 unit 4m 2s hadoop-auth in the patch passed with JDK v1.7.0_95.
          -1 unit 72m 29s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          -1 asflicense 0m 29s Patch generated 1 ASF License warnings.
          228m 0s



          Reason Tests
          JDK v1.8.0_72 Failed junit tests hadoop.hdfs.server.namenode.ha.TestEditLogTailer
            hadoop.hdfs.security.TestDelegationTokenForProxyUser
            hadoop.hdfs.TestFileAppend
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
          JDK v1.8.0_72 Timed out junit tests org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160
            org.apache.hadoop.hdfs.TestReplication
            org.apache.hadoop.hdfs.TestDecommission
            org.apache.hadoop.hdfs.TestDFSStripedOutputStream
          JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.blockmanagement.TestBlockManager
            hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency
            hadoop.hdfs.TestRollingUpgrade
            hadoop.hdfs.server.datanode.TestDataNodeMXBean
          JDK v1.7.0_95 Timed out junit tests org.apache.hadoop.cli.TestHDFSCLI



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12791006/HDFS-9888.01.patch
          JIRA Issue HDFS-9888
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 349d023a40c5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 67880cc
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14694/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14694/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14694/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14694/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14694/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/14694/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-common-project/hadoop-auth hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14694/console
          Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 37s Maven dependency ordering for branch +1 mvninstall 8m 16s trunk passed +1 compile 11m 13s trunk passed with JDK v1.8.0_72 +1 compile 8m 30s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 8s trunk passed +1 mvnsite 1m 25s trunk passed +1 mvneclipse 0m 27s trunk passed +1 findbugs 2m 37s trunk passed +1 javadoc 1m 40s trunk passed with JDK v1.8.0_72 +1 javadoc 2m 21s trunk passed with JDK v1.7.0_95 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 1m 13s the patch passed +1 compile 10m 26s the patch passed with JDK v1.8.0_72 +1 javac 10m 26s the patch passed +1 compile 8m 55s the patch passed with JDK v1.7.0_95 +1 javac 8m 55s the patch passed +1 checkstyle 1m 9s the patch passed +1 mvnsite 1m 25s the patch passed +1 mvneclipse 0m 30s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 3m 5s the patch passed +1 javadoc 1m 42s the patch passed with JDK v1.8.0_72 +1 javadoc 2m 27s the patch passed with JDK v1.7.0_95 +1 unit 3m 58s hadoop-auth in the patch passed with JDK v1.8.0_72. -1 unit 74m 40s hadoop-hdfs in the patch failed with JDK v1.8.0_72. +1 unit 4m 2s hadoop-auth in the patch passed with JDK v1.7.0_95. -1 unit 72m 29s hadoop-hdfs in the patch failed with JDK v1.7.0_95. -1 asflicense 0m 29s Patch generated 1 ASF License warnings. 228m 0s Reason Tests JDK v1.8.0_72 Failed junit tests hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.TestFileAppend   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure JDK v1.8.0_72 Timed out junit tests org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160   org.apache.hadoop.hdfs.TestReplication   org.apache.hadoop.hdfs.TestDecommission   org.apache.hadoop.hdfs.TestDFSStripedOutputStream JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.blockmanagement.TestBlockManager   hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency   hadoop.hdfs.TestRollingUpgrade   hadoop.hdfs.server.datanode.TestDataNodeMXBean JDK v1.7.0_95 Timed out junit tests org.apache.hadoop.cli.TestHDFSCLI Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12791006/HDFS-9888.01.patch JIRA Issue HDFS-9888 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 349d023a40c5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 67880cc Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/14694/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14694/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14694/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14694/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14694/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/14694/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-common-project/hadoop-auth hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14694/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          The test failure and license warning looks unrelated. Zhe Zhang, could you see if my above comment and stacktrace make sense? Thanks.

          Show
          xiaochen Xiao Chen added a comment - The test failure and license warning looks unrelated. Zhe Zhang , could you see if my above comment and stacktrace make sense? Thanks.
          Hide
          zhz Zhe Zhang added a comment -

          Thanks Xiao for the analysis! It makes sense and the patch LGTM. +1 and I'll commit shortly.

          Just curious, do you know when HDFS-3016 didn't get into branch-2 and should we port it to branch-2?

          Show
          zhz Zhe Zhang added a comment - Thanks Xiao for the analysis! It makes sense and the patch LGTM. +1 and I'll commit shortly. Just curious, do you know when HDFS-3016 didn't get into branch-2 and should we port it to branch-2?
          Hide
          xiaochen Xiao Chen added a comment -

          Yes Zhe, HDFS-3016 is not in branch-2.
          Additionally, since TestBalancer#testBalancerWithKeytabs is added by HDFS-9804 which is not in branch-2, I think trunk is enough for this patch. I've added a link to that jira, so that if later we're backporting it, we'll bring this as well. Thanks!

          Show
          xiaochen Xiao Chen added a comment - Yes Zhe, HDFS-3016 is not in branch-2. Additionally, since TestBalancer#testBalancerWithKeytabs is added by HDFS-9804 which is not in branch-2, I think trunk is enough for this patch. I've added a link to that jira, so that if later we're backporting it, we'll bring this as well. Thanks!
          Hide
          zhz Zhe Zhang added a comment -

          Thanks Xiao. I just committed the patch to trunk. Also updated the JIRA title since we added a public method to non-test code.

          Show
          zhz Zhe Zhang added a comment - Thanks Xiao. I just committed the patch to trunk. Also updated the JIRA title since we added a public method to non-test code.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9424 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9424/)
          HDFS-9888. Allow reseting KerberosName in unit tests. Contributed by (zhz: rev 3e8099a45a4cfd4c5c0e3dce4370514cb2c90da9)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
          • hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/KerberosName.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9424 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9424/ ) HDFS-9888 . Allow reseting KerberosName in unit tests. Contributed by (zhz: rev 3e8099a45a4cfd4c5c0e3dce4370514cb2c90da9) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/KerberosName.java
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks very much Zhe Zhang!

          Show
          xiaochen Xiao Chen added a comment - Thanks very much Zhe Zhang !
          Hide
          xiaochen Xiao Chen added a comment -

          FYI, I just noticed that the commit to trunk has a bunch of whitespace changes in both files. Nothing functionally different than patch 01 here though.

          Show
          xiaochen Xiao Chen added a comment - FYI, I just noticed that the commit to trunk has a bunch of whitespace changes in both files. Nothing functionally different than patch 01 here though.
          Hide
          zhz Zhe Zhang added a comment -

          Sorry, I changed laptop around that time. Must be a bad IntelliJ setting at that time.

          Show
          zhz Zhe Zhang added a comment - Sorry, I changed laptop around that time. Must be a bad IntelliJ setting at that time.
          Hide
          zhz Zhe Zhang added a comment -

          I just backported this patch to branch-2, branch-2.8 and branch-2.7.

          Show
          zhz Zhe Zhang added a comment - I just backported this patch to branch-2, branch-2.8 and branch-2.7.

            People

            • Assignee:
              xiaochen Xiao Chen
              Reporter:
              xiaochen Xiao Chen
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development