Hadoop Common
  1. Hadoop Common
  2. HADOOP-8251

SecurityUtil.fetchServiceTicket broken after HADOOP-6941

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 1.1.0, 2.0.0-alpha
    • Fix Version/s: 1.0.3, 2.0.0-alpha, 0.23.7
    • Component/s: security
    • Labels:
      None

      Description

      HADOOP-6941 replaced direct references to some classes with reflective access so as to support other JDKs. Unfortunately there was a mistake in the name of the Krb5Util class, which broke fetchServiceTicket. This manifests itself as the inability to run checkpoints or other krb5-SSL HTTP-based transfers:

      java.lang.ClassNotFoundException: sun.security.jgss.krb5

      1. hadoop-8251.txt
        0.9 kB
        Todd Lipcon
      2. hadoop-8251-b1.txt
        0.7 kB
        Todd Lipcon

        Issue Links

          Activity

          Hide
          Todd Lipcon added a comment -

          The bug was simple – the string used for the name of the Krb5Util class was mistakenly just the package name instead of the class name.

          It looks like the IBM implementation has the same bug, but googling around, I don't think there even is a Krb5Util class in IBM's library, at least not with the functions we need. So I am skeptical that security support works when running on the IBM JRE.

          Show
          Todd Lipcon added a comment - The bug was simple – the string used for the name of the Krb5Util class was mistakenly just the package name instead of the class name. It looks like the IBM implementation has the same bug, but googling around, I don't think there even is a Krb5Util class in IBM's library, at least not with the functions we need. So I am skeptical that security support works when running on the IBM JRE.
          Hide
          Eli Collins added a comment -

          +1 nice catch Todd. HADOOP-7211 covers security and the IBM jdk.

          Show
          Eli Collins added a comment - +1 nice catch Todd. HADOOP-7211 covers security and the IBM jdk.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12521640/hadoop-8251.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.fs.viewfs.TestViewFsTrash

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/829//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/829//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12521640/hadoop-8251.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.fs.viewfs.TestViewFsTrash +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/829//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/829//console This message is automatically generated.
          Hide
          Todd Lipcon added a comment -

          Attaching branch-1 patch, which is identical but for the file paths. I'll commit this momentarily

          Show
          Todd Lipcon added a comment - Attaching branch-1 patch, which is identical but for the file paths. I'll commit this momentarily
          Hide
          Todd Lipcon added a comment -

          Committed to branch-1, branch-1.0, branch-2, and trunk (whew!)

          Show
          Todd Lipcon added a comment - Committed to branch-1, branch-1.0, branch-2, and trunk (whew!)
          Hide
          Devaraj Das added a comment -

          Excellent catch, Todd!

          Could you please address the IBM code patch as well. I did a quick check again on the JDK I have:
          java version "1.6.0"
          Java(TM) SE Runtime Environment (build pxi3260sr10-20111208_01(SR10))
          IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 Linux x86-32 jvmxi3260sr10-20111207_96808 (JIT enabled, AOT enabled)
          J9VM - 20111207_096808
          JIT - r9_20111107_21307ifx1
          GC - 20110519_AA)
          JCL - 20111104_02,

          It seems Krb5Util is there. Also, the methods we are interested in seems to be there.

          The jar that contains the Krb5Util.class is ibmjgssprovider.jar in the JDK.

          The output of javap on the extracted class is:

          Compiled from "Krb5Util.java"
          public class com.ibm.security.jgss.mech.krb5.Krb5Util extends java.lang.Object

          { public static javax.security.auth.kerberos.KerberosTicket getTicketFromSubjectAndTgs(int, java.lang.String, java.lang.String, java.lang.String, java.security.AccessControlContext) throws javax.security.auth.login.LoginException, com.ibm.security.krb5.KrbException, java.io.IOException; static javax.security.auth.kerberos.KerberosTicket getTicket(int, java.lang.String, java.lang.String, java.security.AccessControlContext) throws javax.security.auth.login.LoginException; public static javax.security.auth.Subject getSubject(int, java.security.AccessControlContext) throws javax.security.auth.login.LoginException; public static javax.security.auth.kerberos.KerberosKey[] getKeys(int, java.lang.String, java.security.AccessControlContext) throws javax.security.auth.login.LoginException; public static javax.security.auth.kerberos.KerberosTicket credsToTicket(com.ibm.security.krb5.Credentials); public static com.ibm.security.krb5.Credentials ticketToCreds(javax.security.auth.kerberos.KerberosTicket) throws com.ibm.security.krb5.KrbException, java.io.IOException; }

          Seems like the methods are there and with the desired signatures..

          Did I miss something?

          Show
          Devaraj Das added a comment - Excellent catch, Todd! Could you please address the IBM code patch as well. I did a quick check again on the JDK I have: java version "1.6.0" Java(TM) SE Runtime Environment (build pxi3260sr10-20111208_01(SR10)) IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 Linux x86-32 jvmxi3260sr10-20111207_96808 (JIT enabled, AOT enabled) J9VM - 20111207_096808 JIT - r9_20111107_21307ifx1 GC - 20110519_AA) JCL - 20111104_02, It seems Krb5Util is there. Also, the methods we are interested in seems to be there. The jar that contains the Krb5Util.class is ibmjgssprovider.jar in the JDK. The output of javap on the extracted class is: Compiled from "Krb5Util.java" public class com.ibm.security.jgss.mech.krb5.Krb5Util extends java.lang.Object { public static javax.security.auth.kerberos.KerberosTicket getTicketFromSubjectAndTgs(int, java.lang.String, java.lang.String, java.lang.String, java.security.AccessControlContext) throws javax.security.auth.login.LoginException, com.ibm.security.krb5.KrbException, java.io.IOException; static javax.security.auth.kerberos.KerberosTicket getTicket(int, java.lang.String, java.lang.String, java.security.AccessControlContext) throws javax.security.auth.login.LoginException; public static javax.security.auth.Subject getSubject(int, java.security.AccessControlContext) throws javax.security.auth.login.LoginException; public static javax.security.auth.kerberos.KerberosKey[] getKeys(int, java.lang.String, java.security.AccessControlContext) throws javax.security.auth.login.LoginException; public static javax.security.auth.kerberos.KerberosTicket credsToTicket(com.ibm.security.krb5.Credentials); public static com.ibm.security.krb5.Credentials ticketToCreds(javax.security.auth.kerberos.KerberosTicket) throws com.ibm.security.krb5.KrbException, java.io.IOException; } Seems like the methods are there and with the desired signatures.. Did I miss something?
          Hide
          Todd Lipcon added a comment -

          Hey Devaraj. Sorry, I already committed this, and I don't feel comfortable changing the code if I can't test it (I don't have ready access to an IBM JDK installation). I think rather than just fixing this bug, someone should run through the whole security test plan on the IBM JDK – perhaps as part of HADOOP-7211?

          Seems like the methods are there and with the desired signatures..

          Did I miss something?

          I was basing it on these docs:
          http://www.ibm.com/developerworks/java/jdk/security/60/secguides/jgssDocs/api/index.html?com/ibm/security/jgss/mech/krb5/Krb5RealmUtil.html
          which don't mention krb5util in that package

          Show
          Todd Lipcon added a comment - Hey Devaraj. Sorry, I already committed this, and I don't feel comfortable changing the code if I can't test it (I don't have ready access to an IBM JDK installation). I think rather than just fixing this bug, someone should run through the whole security test plan on the IBM JDK – perhaps as part of HADOOP-7211 ? Seems like the methods are there and with the desired signatures.. Did I miss something? I was basing it on these docs: http://www.ibm.com/developerworks/java/jdk/security/60/secguides/jgssDocs/api/index.html?com/ibm/security/jgss/mech/krb5/Krb5RealmUtil.html which don't mention krb5util in that package
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2087 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2087/)
          HADOOP-8251. Fix SecurityUtil.fetchServiceTicket after HADOOP-6941. Contributed by Todd Lipcon. (Revision 1310168)

          Result = SUCCESS
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1310168
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2087 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2087/ ) HADOOP-8251 . Fix SecurityUtil.fetchServiceTicket after HADOOP-6941 . Contributed by Todd Lipcon. (Revision 1310168) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1310168 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2025 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2025/)
          HADOOP-8251. Fix SecurityUtil.fetchServiceTicket after HADOOP-6941. Contributed by Todd Lipcon. (Revision 1310168)

          Result = SUCCESS
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1310168
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2025 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2025/ ) HADOOP-8251 . Fix SecurityUtil.fetchServiceTicket after HADOOP-6941 . Contributed by Todd Lipcon. (Revision 1310168) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1310168 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2012 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2012/)
          HADOOP-8251. Fix SecurityUtil.fetchServiceTicket after HADOOP-6941. Contributed by Todd Lipcon. (Revision 1310168)

          Result = SUCCESS
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1310168
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2012 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2012/ ) HADOOP-8251 . Fix SecurityUtil.fetchServiceTicket after HADOOP-6941 . Contributed by Todd Lipcon. (Revision 1310168) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1310168 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
          Hide
          Luke Lu added a comment -

          Krb5Util is available in J9 SR10 (the current version of IBM JDK). I thought that I saw a TestSecurityUtil in ddas' patch. I guess not

          Please have at least one simple test that fails without the patch.

          Show
          Luke Lu added a comment - Krb5Util is available in J9 SR10 (the current version of IBM JDK). I thought that I saw a TestSecurityUtil in ddas' patch. I guess not Please have at least one simple test that fails without the patch.
          Hide
          Todd Lipcon added a comment -

          Please have at least one simple test that fails without the patch.

          I'm just fixing what the previous patch broke. I don't have time to write a test, since this depends on security infrastructure, etc, and I can't get that to work right (see my comment on HDFS-3016). The original patch should have had a test, I agree. But my options were to revert that patch, or just fix it, so I did the latter without a test.

          Show
          Todd Lipcon added a comment - Please have at least one simple test that fails without the patch. I'm just fixing what the previous patch broke. I don't have time to write a test, since this depends on security infrastructure, etc, and I can't get that to work right (see my comment on HDFS-3016 ). The original patch should have had a test, I agree. But my options were to revert that patch, or just fix it, so I did the latter without a test.
          Hide
          Devaraj Das added a comment -

          Sorry, I already committed this, and I don't feel comfortable changing the code if I can't test it (I don't have ready access to an IBM JDK installation). I think rather than just fixing this bug, someone should run through the whole security test plan on the IBM JDK – perhaps as part of HADOOP-7211?

          Fair point. Works for me.

          Show
          Devaraj Das added a comment - Sorry, I already committed this, and I don't feel comfortable changing the code if I can't test it (I don't have ready access to an IBM JDK installation). I think rather than just fixing this bug, someone should run through the whole security test plan on the IBM JDK – perhaps as part of HADOOP-7211 ? Fair point. Works for me.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1006 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1006/)
          HADOOP-8251. Fix SecurityUtil.fetchServiceTicket after HADOOP-6941. Contributed by Todd Lipcon. (Revision 1310168)

          Result = FAILURE
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1310168
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1006 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1006/ ) HADOOP-8251 . Fix SecurityUtil.fetchServiceTicket after HADOOP-6941 . Contributed by Todd Lipcon. (Revision 1310168) Result = FAILURE todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1310168 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1041 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1041/)
          HADOOP-8251. Fix SecurityUtil.fetchServiceTicket after HADOOP-6941. Contributed by Todd Lipcon. (Revision 1310168)

          Result = SUCCESS
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1310168
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1041 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1041/ ) HADOOP-8251 . Fix SecurityUtil.fetchServiceTicket after HADOOP-6941 . Contributed by Todd Lipcon. (Revision 1310168) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1310168 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
          Hide
          Matt Foley added a comment -

          Closed upon release of Hadoop-1.0.3.

          Show
          Matt Foley added a comment - Closed upon release of Hadoop-1.0.3.
          Hide
          Matt Foley added a comment - - edited

          1.0.3 was a lineal predecessor of 1.1.0.

          Show
          Matt Foley added a comment - - edited 1.0.3 was a lineal predecessor of 1.1.0.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #512 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/512/)
          HADOOP-8251. Fix SecurityUtil.fetchServiceTicket after HADOOP-6941 (todd via tgraves) (Revision 1441230)

          Result = SUCCESS
          tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1441230
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #512 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/512/ ) HADOOP-8251 . Fix SecurityUtil.fetchServiceTicket after HADOOP-6941 (todd via tgraves) (Revision 1441230) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1441230 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java

            People

            • Assignee:
              Todd Lipcon
              Reporter:
              Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development