Details

    • Hadoop Flags:
      Reviewed

      Description

      Krb5LoginModule changed subtly in java 8: in particular, if useKeyTab and storeKey are specified, then only a KeyTab object is added to the Subject's private credentials, whereas in java <= 7 both a KeyTab and some number of KerberosKey objects were added.

      The UGI constructor checks whether or not a keytab was used to login by looking if there are any KerberosKey objects in the Subject's private credentials. If there are, then isKeyTab is set to true, and otherwise it's set to false.

      Thus, in java 8 isKeyTab is always false given the current UGI implementation, which makes UGI#reloginFromKeytab fail silently.

      Attached patch will check for a KeyTab object on the Subject, instead of a KerberosKey object. This fixes relogins from kerberos keytabs on Oracle java 8, and works on Oracle java 7 as well.

      1. HADOOP-10786.2.patch
        7 kB
        Stephen Chu
      2. HADOOP-10786.3.patch
        7 kB
        Stephen Chu
      3. HADOOP-10786.3.patch
        7 kB
        Stephen Chu
      4. HADOOP-10786.4.patch
        7 kB
        Stephen Chu
      5. HADOOP-10786.5.patch
        7 kB
        Stephen Chu
      6. HADOOP-10786.patch
        1 kB
        Tobi Vollebregt

        Issue Links

          Activity

          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12654166/HADOOP-10786.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 javac. The patch appears to cause the build to fail.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4217//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12654166/HADOOP-10786.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javac . The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4217//console This message is automatically generated.
          Hide
          Tobi Tobi Vollebregt added a comment -

          Not sure why build is failing, there are no compile errors in the log. Can't reproduce build failure locally - build is failing (flaky?) even on trunk.

          Show
          Tobi Tobi Vollebregt added a comment - Not sure why build is failing, there are no compile errors in the log. Can't reproduce build failure locally - build is failing (flaky?) even on trunk.
          Hide
          cwsteinbach Carl Steinbach added a comment -

          Tobi Vollebregt, thanks for sharing this patch. We encountered the same problem on a Hadoop 2.3.0 cluster and were able to resolve it with this change.

          Show
          cwsteinbach Carl Steinbach added a comment - Tobi Vollebregt , thanks for sharing this patch. We encountered the same problem on a Hadoop 2.3.0 cluster and were able to resolve it with this change.
          Hide
          stevel@apache.org Steve Loughran added a comment -
          1. what does this do on Java 6?
          2. It may actually be possible to write a test for this with MiniKDC; this is clearly something that's not being tested for today. Adding that test would would ensure we don't regress again
          Show
          stevel@apache.org Steve Loughran added a comment - what does this do on Java 6? It may actually be possible to write a test for this with MiniKDC; this is clearly something that's not being tested for today. Adding that test would would ensure we don't regress again
          Hide
          stevel@apache.org Steve Loughran added a comment -

          keytab is tagged as {{ * @ since 1.7 }}, so can't go in while Hadoop is still built against Java 6.

          Show
          stevel@apache.org Steve Loughran added a comment - keytab is tagged as {{ * @ since 1.7 }}, so can't go in while Hadoop is still built against Java 6.
          Hide
          apurtell Andrew Purtell added a comment -

          Shouldn't this be a higher priority than 'Minor'? The end of public updates to Java 7 will be April 2015. A silent failure to re-login from keytab after TGT expiry dooms any long running process that wants to use secure RPC. Anyone who cares about security and about running the best performing supported Java runtime shortly will be forced to locally patch their core libraries.

          Show
          apurtell Andrew Purtell added a comment - Shouldn't this be a higher priority than 'Minor'? The end of public updates to Java 7 will be April 2015. A silent failure to re-login from keytab after TGT expiry dooms any long running process that wants to use secure RPC. Anyone who cares about security and about running the best performing supported Java runtime shortly will be forced to locally patch their core libraries.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12654166/HADOOP-10786.patch
          against trunk revision c0c7e6f.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 javac. The patch appears to cause the build to fail.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4747//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12654166/HADOOP-10786.patch against trunk revision c0c7e6f. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javac . The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4747//console This message is automatically generated.
          Hide
          schu Stephen Chu added a comment -

          We can use reflection to make this fix and still allow JDK6 to build and run.

          I've attached a patch to do this, as well as added a unit test that will catch regressions. The unit test uses the MiniKDC and verifies login from keytab and relogin from keytab in addition to simply checking that isKeytab = true when it should be.

          Tobi Vollebregt, thanks a lot for working on this. Let me know what you think about my suggestion and test. If you are too busy, I can also take this JIRA up.

          Show
          schu Stephen Chu added a comment - We can use reflection to make this fix and still allow JDK6 to build and run. I've attached a patch to do this, as well as added a unit test that will catch regressions. The unit test uses the MiniKDC and verifies login from keytab and relogin from keytab in addition to simply checking that isKeytab = true when it should be. Tobi Vollebregt , thanks a lot for working on this. Let me know what you think about my suggestion and test. If you are too busy, I can also take this JIRA up.
          Hide
          schu Stephen Chu added a comment -

          Note that the test reproduces the issue when run against JDK8 on a build without the fix.

          I then built and ran successfully with the fix for JDK 6, 7, and 8.

          Show
          schu Stephen Chu added a comment - Note that the test reproduces the issue when run against JDK8 on a build without the fix. I then built and ran successfully with the fix for JDK 6, 7, and 8.
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12679752/HADOOP-10786.2.patch
          against trunk revision 6ba52d8.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5039//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5039//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12679752/HADOOP-10786.2.patch against trunk revision 6ba52d8. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5039//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5039//console This message is automatically generated.
          Hide
          brocknoland Brock Noland added a comment -

          That's one clean patch.

          Show
          brocknoland Brock Noland added a comment - That's one clean patch.
          Hide
          Tobi Tobi Vollebregt added a comment -

          Thank you Stephen Chu, that's a great patch. What else would need to be done to land this?

          Show
          Tobi Tobi Vollebregt added a comment - Thank you Stephen Chu , that's a great patch. What else would need to be done to land this?
          Hide
          schu Stephen Chu added a comment -

          Thanks, Tobi. We just need to wait for reviews and make any iterations that come from the reviews. When a committer +1's, then the approved patch can be committed.

          Show
          schu Stephen Chu added a comment - Thanks, Tobi. We just need to wait for reviews and make any iterations that come from the reviews. When a committer +1's, then the approved patch can be committed.
          Hide
          wheat9 Haohui Mai added a comment -
          +    try {
          +      // In JDK6 and JDK7, if useKeyTab and storeKey are specified in the
          +      // Krb5LoginModule, then some number of KerberosKey objects are added
          +      // to the Subject's private credentials. However, in JDK8, a KeyTab
          +      // object is added instead. More details in HADOOP-10786.
          +      keytabClass = Class.forName("javax.security.auth.kerberos.KeyTab");
          +    } catch (ClassNotFoundException cnfe) {
          +      // Ignore. javax.security.auth.kerberos.KeyTab does not exist in JDK6.
          +    }
          +    if (keytabClass != null) {
          +      this.isKeytab = !subject.getPrivateCredentials(keytabClass).isEmpty();
          +    } else {
          +      this.isKeytab = !subject.getPrivateCredentials(KerberosKey.class).isEmpty();
          +    }
               this.isKrbTkt = !subject.getPrivateCredentials(KerberosTicket.class).isEmpty();
          

          forName is fairly slow. Since the patch is targeting 2.7 which only supports JDK7, the code should be able to use the class in compile time, though we'll need to wait until jenkins to be switched to Java 7 before this patch can land.

          +  @VisibleForTesting
          +  static void setShouldRenewImmediatelyForTests(boolean immediate) {
          +    shouldRenewImmediatelyForTests = immediate;
          +  }
          

          Instead of adding this method, it might make more sense to extract the logic of login into a separate function and call the function directly in the tests.

          Show
          wheat9 Haohui Mai added a comment - + try { + // In JDK6 and JDK7, if useKeyTab and storeKey are specified in the + // Krb5LoginModule, then some number of KerberosKey objects are added + // to the Subject's private credentials. However, in JDK8, a KeyTab + // object is added instead. More details in HADOOP-10786. + keytabClass = Class .forName( "javax.security.auth.kerberos.KeyTab" ); + } catch (ClassNotFoundException cnfe) { + // Ignore. javax.security.auth.kerberos.KeyTab does not exist in JDK6. + } + if (keytabClass != null ) { + this .isKeytab = !subject.getPrivateCredentials(keytabClass).isEmpty(); + } else { + this .isKeytab = !subject.getPrivateCredentials(KerberosKey.class).isEmpty(); + } this .isKrbTkt = !subject.getPrivateCredentials(KerberosTicket.class).isEmpty(); forName is fairly slow. Since the patch is targeting 2.7 which only supports JDK7, the code should be able to use the class in compile time, though we'll need to wait until jenkins to be switched to Java 7 before this patch can land. + @VisibleForTesting + static void setShouldRenewImmediatelyForTests( boolean immediate) { + shouldRenewImmediatelyForTests = immediate; + } Instead of adding this method, it might make more sense to extract the logic of login into a separate function and call the function directly in the tests.
          Hide
          atm Aaron T. Myers added a comment - - edited

          forName is fairly slow. Since the patch is targeting 2.7 which only supports JDK7, the code should be able to use the class in compile time, though we'll need to wait until jenkins to be switched to Java 7 before this patch can land.

          I'd like to commit this for 2.6, though, which will still be targeting Java 6. How about we create a constant KEY_TAB_CLASS and then do the reflection in a static initialization block? That way we only pay the lookup penalty once per JVM and the patch still works with both version of Java.

          Show
          atm Aaron T. Myers added a comment - - edited forName is fairly slow. Since the patch is targeting 2.7 which only supports JDK7, the code should be able to use the class in compile time, though we'll need to wait until jenkins to be switched to Java 7 before this patch can land. I'd like to commit this for 2.6, though, which will still be targeting Java 6. How about we create a constant KEY_TAB_CLASS and then do the reflection in a static initialization block? That way we only pay the lookup penalty once per JVM and the patch still works with both version of Java.
          Hide
          schu Stephen Chu added a comment -

          Performing the reflection in a static init block sounds like a good idea.

          I can see how it'd be useful to extract the logic of login into a separate function and just call it directly. I'd like to make sure to exercise as much of the reloginFromKeytab logic as possible (aside from waiting for a renew window), though.

          The test verifies isKeytab == true, which is good. However, if for some reason the way isKeytab changes in reloginFromKeytab (or something else changes before actual login), it'd be good to exercise this.

          Attaching a patch that moves the reflection to a static block.

          Also, I made some additional fixes:

          • Fix the conditional logic when using shouldRenewImmediatelyForTests by moving the check for null TGT ahead.
          • Remove //return
          Show
          schu Stephen Chu added a comment - Performing the reflection in a static init block sounds like a good idea. I can see how it'd be useful to extract the logic of login into a separate function and just call it directly. I'd like to make sure to exercise as much of the reloginFromKeytab logic as possible (aside from waiting for a renew window), though. The test verifies isKeytab == true, which is good. However, if for some reason the way isKeytab changes in reloginFromKeytab (or something else changes before actual login), it'd be good to exercise this. Attaching a patch that moves the reflection to a static block. Also, I made some additional fixes: Fix the conditional logic when using shouldRenewImmediatelyForTests by moving the check for null TGT ahead. Remove //return
          Hide
          schu Stephen Chu added a comment -

          Resubmitting HADOOP-10786.3.patch to hopefully retrigger Hadoop QA jenkins.

          Show
          schu Stephen Chu added a comment - Resubmitting HADOOP-10786 .3.patch to hopefully retrigger Hadoop QA jenkins.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12680235/HADOOP-10786.3.patch
          against trunk revision 2ac1be7.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common:

          org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5046//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5046//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12680235/HADOOP-10786.3.patch against trunk revision 2ac1be7. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common: org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5046//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5046//console This message is automatically generated.
          Hide
          schu Stephen Chu added a comment -

          TestMetricsSystemImpl is unrelated. I ran the test locally and successfully with the patch. Outstanding JIRA for it at HADOOP-10062.

          Show
          schu Stephen Chu added a comment - TestMetricsSystemImpl is unrelated. I ran the test locally and successfully with the patch. Outstanding JIRA for it at HADOOP-10062 .
          Hide
          schu Stephen Chu added a comment -

          Posting a new patch to fix a mistake. Thanks to ATM for catching it.

          If the following conditional is true, we should return:

          if (tgt != null && !shouldRenewImmediatelyForTests &&
                  now < getRefreshTime(tgt)) {
          

          I had forgotten to clean up some debugging stuff.

          Show
          schu Stephen Chu added a comment - Posting a new patch to fix a mistake. Thanks to ATM for catching it. If the following conditional is true, we should return: if (tgt != null && !shouldRenewImmediatelyForTests && now < getRefreshTime(tgt)) { I had forgotten to clean up some debugging stuff.
          Hide
          schu Stephen Chu added a comment -

          Oops, wrong patch. Cancelling and will resubmit.

          Show
          schu Stephen Chu added a comment - Oops, wrong patch. Cancelling and will resubmit.
          Hide
          schu Stephen Chu added a comment -

          Note that the correct patch is HADOOP-10786.4.patch.

          Show
          schu Stephen Chu added a comment - Note that the correct patch is HADOOP-10786 .4.patch.
          Hide
          atm Aaron T. Myers added a comment -

          Thanks a lot, Stephen. The latest patch looks good to me. I'll be +1 on it pending Jenkins.

          Haohui - just checking, has this latest patch addressed your performance concern?

          Thanks folks.

          Show
          atm Aaron T. Myers added a comment - Thanks a lot, Stephen. The latest patch looks good to me. I'll be +1 on it pending Jenkins. Haohui - just checking, has this latest patch addressed your performance concern? Thanks folks.
          Hide
          wheat9 Haohui Mai added a comment -

          Looks pretty good to me. Some nits:

          +  private static Class<?> KEY_TAB_CLASS = KerberosKey.class;
          

          can be

          +  private static final Class<?> KEY_TAB_CLASS = KerberosKey.class;
          
          +  public void createTestDir() {
          +    workDir = new File(System.getProperty("test.dir", "target"));
          +  }
          

          You can use TemporaryFolder here so that the files can be properly cleaned up after the tests.

          +  public File getWorkDir() {
          +    return workDir;
          +  }
          +  public MiniKdc getKdc() {
          +    return kdc;
          +  }
          +
          

          Looks like the test can simply inline these getters to make the patch even smaller.

          Show
          wheat9 Haohui Mai added a comment - Looks pretty good to me. Some nits: + private static Class <?> KEY_TAB_CLASS = KerberosKey.class; can be + private static final Class <?> KEY_TAB_CLASS = KerberosKey.class; + public void createTestDir() { + workDir = new File( System .getProperty( "test.dir" , "target" )); + } You can use TemporaryFolder here so that the files can be properly cleaned up after the tests. + public File getWorkDir() { + return workDir; + } + public MiniKdc getKdc() { + return kdc; + } + Looks like the test can simply inline these getters to make the patch even smaller.
          Hide
          schu Stephen Chu added a comment -

          Thanks for the comments, Haohui. Attaching patch to address your comments.

          KEY_TAB_CLASS will be reassigned to Class.forName("javax.security.auth.kerberos.KeyTab") on JDK7/8, so I couldn't declare it final.

          The new patch now uses TemporaryFolder to ensure cleanup of the dir after test run.

          Also removed the unnecessary getters.

          Show
          schu Stephen Chu added a comment - Thanks for the comments, Haohui. Attaching patch to address your comments. KEY_TAB_CLASS will be reassigned to Class.forName("javax.security.auth.kerberos.KeyTab") on JDK7/8, so I couldn't declare it final. The new patch now uses TemporaryFolder to ensure cleanup of the dir after test run. Also removed the unnecessary getters.
          Hide
          wheat9 Haohui Mai added a comment -

          Oh I see. The patch looks good to me. +1 pending jenkins.

          Show
          wheat9 Haohui Mai added a comment - Oh I see. The patch looks good to me. +1 pending jenkins.
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12680274/HADOOP-10786.5.patch
          against trunk revision 68db5b3.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5049//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5049//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12680274/HADOOP-10786.5.patch against trunk revision 68db5b3. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5049//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5049//console This message is automatically generated.
          Hide
          wheat9 Haohui Mai added a comment -

          I've committed the patch to trunk, branch-2 and branch-2.6. Thanks Stephen Chu for the contribution.

          Show
          wheat9 Haohui Mai added a comment - I've committed the patch to trunk, branch-2 and branch-2.6. Thanks Stephen Chu for the contribution.
          Hide
          wheat9 Haohui Mai added a comment -

          I created HADOOP-10287 to track the effort of simplifying the code in 2.7.

          Show
          wheat9 Haohui Mai added a comment - I created HADOOP-10287 to track the effort of simplifying the code in 2.7.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #6496 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6496/)
          HADOOP-10786. Fix UGI#reloginFromKeytab on Java 8. Contributed by Stephen Chu. (wheat9: rev a37a993453c02048a618f71b5b9bc63b5a44dbf6)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestUGILoginFromKeytab.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #6496 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6496/ ) HADOOP-10786 . Fix UGI#reloginFromKeytab on Java 8. Contributed by Stephen Chu. (wheat9: rev a37a993453c02048a618f71b5b9bc63b5a44dbf6) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestUGILoginFromKeytab.java
          Hide
          acmurthy Arun C Murthy added a comment -

          I think it's too late/risky to put this into 2.6; let's get this into 2.7. Thanks.

          Show
          acmurthy Arun C Murthy added a comment - I think it's too late/risky to put this into 2.6; let's get this into 2.7. Thanks.
          Hide
          acmurthy Arun C Murthy added a comment -

          Moved to hadoop-2.7.0.

          Show
          acmurthy Arun C Murthy added a comment - Moved to hadoop-2.7.0.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #6498 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6498/)
          HADOOP-10786. Moved to hadoop-2.7.X. (acmurthy: rev 14b87b70a8dfc03801dcf5f33caa7fd2cc589840)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #6498 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6498/ ) HADOOP-10786 . Moved to hadoop-2.7.X. (acmurthy: rev 14b87b70a8dfc03801dcf5f33caa7fd2cc589840) hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #739 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/739/)
          HADOOP-10786. Fix UGI#reloginFromKeytab on Java 8. Contributed by Stephen Chu. (wheat9: rev a37a993453c02048a618f71b5b9bc63b5a44dbf6)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestUGILoginFromKeytab.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
            HADOOP-10786. Moved to hadoop-2.7.X. (acmurthy: rev 14b87b70a8dfc03801dcf5f33caa7fd2cc589840)
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #739 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/739/ ) HADOOP-10786 . Fix UGI#reloginFromKeytab on Java 8. Contributed by Stephen Chu. (wheat9: rev a37a993453c02048a618f71b5b9bc63b5a44dbf6) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestUGILoginFromKeytab.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java HADOOP-10786 . Moved to hadoop-2.7.X. (acmurthy: rev 14b87b70a8dfc03801dcf5f33caa7fd2cc589840) hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #1929 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1929/)
          HADOOP-10786. Fix UGI#reloginFromKeytab on Java 8. Contributed by Stephen Chu. (wheat9: rev a37a993453c02048a618f71b5b9bc63b5a44dbf6)

          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestUGILoginFromKeytab.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
            HADOOP-10786. Moved to hadoop-2.7.X. (acmurthy: rev 14b87b70a8dfc03801dcf5f33caa7fd2cc589840)
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #1929 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1929/ ) HADOOP-10786 . Fix UGI#reloginFromKeytab on Java 8. Contributed by Stephen Chu. (wheat9: rev a37a993453c02048a618f71b5b9bc63b5a44dbf6) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestUGILoginFromKeytab.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java HADOOP-10786 . Moved to hadoop-2.7.X. (acmurthy: rev 14b87b70a8dfc03801dcf5f33caa7fd2cc589840) hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #1953 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1953/)
          HADOOP-10786. Fix UGI#reloginFromKeytab on Java 8. Contributed by Stephen Chu. (wheat9: rev a37a993453c02048a618f71b5b9bc63b5a44dbf6)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestUGILoginFromKeytab.java
            HADOOP-10786. Moved to hadoop-2.7.X. (acmurthy: rev 14b87b70a8dfc03801dcf5f33caa7fd2cc589840)
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1953 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1953/ ) HADOOP-10786 . Fix UGI#reloginFromKeytab on Java 8. Contributed by Stephen Chu. (wheat9: rev a37a993453c02048a618f71b5b9bc63b5a44dbf6) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestUGILoginFromKeytab.java HADOOP-10786 . Moved to hadoop-2.7.X. (acmurthy: rev 14b87b70a8dfc03801dcf5f33caa7fd2cc589840) hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          cnauroth Chris Nauroth added a comment -

          Stephen Chu and Tobi Vollebregt, thank you for providing this patch.

          I just wanted to share with everyone that even though this bug was reported against a JDK 8 code change, it appears the same change has entered the JDK 7 code line. I am seeing the same problem in the most recent OpenJDK build. With JDK 1.7.0_79, I could not repro the problem. After upgrading to JDK 1.7.0_85, I could repro the problem. I don't know the exact minor version number within JDK 7 that first introduced the change, but it's somewhere in that range.

          I also confirmed that this patch fixes the problem for JDK 1.7.0_85 too. To verify, I ran the test without the corresponding fixes in UserGroupInformation. I observed that the test failed on the assertion for ugi.isFromKeytab(). Then, I applied the UserGroupInformation part of the patch, and the test passed.

          Bottom line: If you want to run a secured Hadoop cluster on JDK 1.7.0_85 or later, then you must run Apache Hadoop 2.7.0 or later.

          Show
          cnauroth Chris Nauroth added a comment - Stephen Chu and Tobi Vollebregt , thank you for providing this patch. I just wanted to share with everyone that even though this bug was reported against a JDK 8 code change, it appears the same change has entered the JDK 7 code line. I am seeing the same problem in the most recent OpenJDK build. With JDK 1.7.0_79, I could not repro the problem. After upgrading to JDK 1.7.0_85, I could repro the problem. I don't know the exact minor version number within JDK 7 that first introduced the change, but it's somewhere in that range. I also confirmed that this patch fixes the problem for JDK 1.7.0_85 too. To verify, I ran the test without the corresponding fixes in UserGroupInformation . I observed that the test failed on the assertion for ugi.isFromKeytab() . Then, I applied the UserGroupInformation part of the patch, and the test passed. Bottom line: If you want to run a secured Hadoop cluster on JDK 1.7.0_85 or later, then you must run Apache Hadoop 2.7.0 or later.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Cherry-picked to 2.6.1

          Show
          vinayrpet Vinayakumar B added a comment - Cherry-picked to 2.6.1
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #8300 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8300/)
          HADOOP-10786. Fix UGI#reloginFromKeytab on Java 8. Contributed by Stephen Chu. (vinayakumarb: rev 24a11e39960696d75e58df912ec6aa7283be194d)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #8300 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8300/ ) HADOOP-10786 . Fix UGI#reloginFromKeytab on Java 8. Contributed by Stephen Chu. (vinayakumarb: rev 24a11e39960696d75e58df912ec6aa7283be194d) hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8301 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8301/)
          HADOOP-10786. Fix UGI#reloginFromKeytab on Java 8. Contributed by Stephen Chu. (vinayakumarb: rev e7aa81394dce61cc96d480e21204263a5f2ed153)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8301 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8301/ ) HADOOP-10786 . Fix UGI#reloginFromKeytab on Java 8. Contributed by Stephen Chu. (vinayakumarb: rev e7aa81394dce61cc96d480e21204263a5f2ed153) hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #287 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/287/)
          HADOOP-10786. Fix UGI#reloginFromKeytab on Java 8. Contributed by Stephen Chu. (vinayakumarb: rev 24a11e39960696d75e58df912ec6aa7283be194d)

          • hadoop-common-project/hadoop-common/CHANGES.txt
            HADOOP-10786. Fix UGI#reloginFromKeytab on Java 8. Contributed by Stephen Chu. (vinayakumarb: rev e7aa81394dce61cc96d480e21204263a5f2ed153)
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #287 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/287/ ) HADOOP-10786 . Fix UGI#reloginFromKeytab on Java 8. Contributed by Stephen Chu. (vinayakumarb: rev 24a11e39960696d75e58df912ec6aa7283be194d) hadoop-common-project/hadoop-common/CHANGES.txt HADOOP-10786 . Fix UGI#reloginFromKeytab on Java 8. Contributed by Stephen Chu. (vinayakumarb: rev e7aa81394dce61cc96d480e21204263a5f2ed153) hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #1017 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1017/)
          HADOOP-10786. Fix UGI#reloginFromKeytab on Java 8. Contributed by Stephen Chu. (vinayakumarb: rev 24a11e39960696d75e58df912ec6aa7283be194d)

          • hadoop-common-project/hadoop-common/CHANGES.txt
            HADOOP-10786. Fix UGI#reloginFromKeytab on Java 8. Contributed by Stephen Chu. (vinayakumarb: rev e7aa81394dce61cc96d480e21204263a5f2ed153)
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1017 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1017/ ) HADOOP-10786 . Fix UGI#reloginFromKeytab on Java 8. Contributed by Stephen Chu. (vinayakumarb: rev 24a11e39960696d75e58df912ec6aa7283be194d) hadoop-common-project/hadoop-common/CHANGES.txt HADOOP-10786 . Fix UGI#reloginFromKeytab on Java 8. Contributed by Stephen Chu. (vinayakumarb: rev e7aa81394dce61cc96d480e21204263a5f2ed153) hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #284 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/284/)
          HADOOP-10786. Fix UGI#reloginFromKeytab on Java 8. Contributed by Stephen Chu. (vinayakumarb: rev 24a11e39960696d75e58df912ec6aa7283be194d)

          • hadoop-common-project/hadoop-common/CHANGES.txt
            HADOOP-10786. Fix UGI#reloginFromKeytab on Java 8. Contributed by Stephen Chu. (vinayakumarb: rev e7aa81394dce61cc96d480e21204263a5f2ed153)
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #284 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/284/ ) HADOOP-10786 . Fix UGI#reloginFromKeytab on Java 8. Contributed by Stephen Chu. (vinayakumarb: rev 24a11e39960696d75e58df912ec6aa7283be194d) hadoop-common-project/hadoop-common/CHANGES.txt HADOOP-10786 . Fix UGI#reloginFromKeytab on Java 8. Contributed by Stephen Chu. (vinayakumarb: rev e7aa81394dce61cc96d480e21204263a5f2ed153) hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2233 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2233/)
          HADOOP-10786. Fix UGI#reloginFromKeytab on Java 8. Contributed by Stephen Chu. (vinayakumarb: rev 24a11e39960696d75e58df912ec6aa7283be194d)

          • hadoop-common-project/hadoop-common/CHANGES.txt
            HADOOP-10786. Fix UGI#reloginFromKeytab on Java 8. Contributed by Stephen Chu. (vinayakumarb: rev e7aa81394dce61cc96d480e21204263a5f2ed153)
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2233 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2233/ ) HADOOP-10786 . Fix UGI#reloginFromKeytab on Java 8. Contributed by Stephen Chu. (vinayakumarb: rev 24a11e39960696d75e58df912ec6aa7283be194d) hadoop-common-project/hadoop-common/CHANGES.txt HADOOP-10786 . Fix UGI#reloginFromKeytab on Java 8. Contributed by Stephen Chu. (vinayakumarb: rev e7aa81394dce61cc96d480e21204263a5f2ed153) hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2214 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2214/)
          HADOOP-10786. Fix UGI#reloginFromKeytab on Java 8. Contributed by Stephen Chu. (vinayakumarb: rev 24a11e39960696d75e58df912ec6aa7283be194d)

          • hadoop-common-project/hadoop-common/CHANGES.txt
            HADOOP-10786. Fix UGI#reloginFromKeytab on Java 8. Contributed by Stephen Chu. (vinayakumarb: rev e7aa81394dce61cc96d480e21204263a5f2ed153)
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2214 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2214/ ) HADOOP-10786 . Fix UGI#reloginFromKeytab on Java 8. Contributed by Stephen Chu. (vinayakumarb: rev 24a11e39960696d75e58df912ec6aa7283be194d) hadoop-common-project/hadoop-common/CHANGES.txt HADOOP-10786 . Fix UGI#reloginFromKeytab on Java 8. Contributed by Stephen Chu. (vinayakumarb: rev e7aa81394dce61cc96d480e21204263a5f2ed153) hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #276 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/276/)
          HADOOP-10786. Fix UGI#reloginFromKeytab on Java 8. Contributed by Stephen Chu. (vinayakumarb: rev 24a11e39960696d75e58df912ec6aa7283be194d)

          • hadoop-common-project/hadoop-common/CHANGES.txt
            HADOOP-10786. Fix UGI#reloginFromKeytab on Java 8. Contributed by Stephen Chu. (vinayakumarb: rev e7aa81394dce61cc96d480e21204263a5f2ed153)
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #276 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/276/ ) HADOOP-10786 . Fix UGI#reloginFromKeytab on Java 8. Contributed by Stephen Chu. (vinayakumarb: rev 24a11e39960696d75e58df912ec6aa7283be194d) hadoop-common-project/hadoop-common/CHANGES.txt HADOOP-10786 . Fix UGI#reloginFromKeytab on Java 8. Contributed by Stephen Chu. (vinayakumarb: rev e7aa81394dce61cc96d480e21204263a5f2ed153) hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          cnauroth Chris Nauroth added a comment -

          Thank you, Vinayakumar B.

          Show
          cnauroth Chris Nauroth added a comment - Thank you, Vinayakumar B .
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Vinayakumar B, please stop committing to 2.6.1 (branch-2.6 or branch-2.6.1), we have a fairly elaborate parallel release-process going on for 2.6.1 and these cherry-picks are disrupting our progress. If you want something added to 2.6.1, please post it in the mailing lists. Thanks.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Vinayakumar B , please stop committing to 2.6.1 (branch-2.6 or branch-2.6.1), we have a fairly elaborate parallel release-process going on for 2.6.1 and these cherry-picks are disrupting our progress. If you want something added to 2.6.1, please post it in the mailing lists. Thanks.
          Hide
          vinayrpet Vinayakumar B added a comment -

          we have a fairly elaborate parallel release-process going on for 2.6.1 and these cherry-picks are disrupting our progress.

          Thanks Vinod Kumar Vavilapalli. I am actually cherry-picking only those which are listed for 2.6.1 here https://wiki.apache.org/hadoop/Release-2.6.1-Working-Notes.
          Anyway I will stop further merges until required.
          -Thanks

          Show
          vinayrpet Vinayakumar B added a comment - we have a fairly elaborate parallel release-process going on for 2.6.1 and these cherry-picks are disrupting our progress. Thanks Vinod Kumar Vavilapalli . I am actually cherry-picking only those which are listed for 2.6.1 here https://wiki.apache.org/hadoop/Release-2.6.1-Working-Notes . Anyway I will stop further merges until required. -Thanks
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          This wasn't originally in 2.6.1, must have been committed to 2.6, which was already 2.6.2. I just committed this to 2.6.1. Ran compilation and TestUGILoginFromKeytab before the push.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - This wasn't originally in 2.6.1, must have been committed to 2.6, which was already 2.6.2. I just committed this to 2.6.1. Ran compilation and TestUGILoginFromKeytab before the push.
          Hide
          taoluo Tao Luo added a comment -

          Looks like the Krb5LoginModule change was backported to JDK 1.7.0_80 as well, in case anyone else hits it.

          Show
          taoluo Tao Luo added a comment - Looks like the Krb5LoginModule change was backported to JDK 1.7.0_80 as well, in case anyone else hits it.
          Hide
          liushaohui Liu Shaohui added a comment -

          Tao Luo
          Thanks for your reminding~ We hit the problem in JDK 1.7.0_80.

          Show
          liushaohui Liu Shaohui added a comment - Tao Luo Thanks for your reminding~ We hit the problem in JDK 1.7.0_80.

            People

            • Assignee:
              schu Stephen Chu
              Reporter:
              Tobi Tobi Vollebregt
            • Votes:
              1 Vote for this issue
              Watchers:
              28 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development