Hadoop Common
  1. Hadoop Common
  2. HADOOP-8883

Anonymous fallback in KerberosAuthenticator is broken

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.3-alpha
    • Fix Version/s: 2.0.3-alpha
    • Component/s: None
    • Labels:
    • Hadoop Flags:
      Reviewed

      Description

      HADOOP-8855 changed KerberosAuthenticator to handle when the JDK did the SPNEGO already; but this change broke using the fallback authenticator (PseudoAuthenticator) with an anonymous user (see OOZIE-1010).

      1. HADOOP-8883.patch
        3 kB
        Robert Kanter

        Issue Links

          Activity

          Hide
          Robert Kanter added a comment -

          JDK SPNEGO puts the "Authorization" header in the request, so we can check that in the if statement that was added in HADOOP-8855.

          Show
          Robert Kanter added a comment - JDK SPNEGO puts the "Authorization" header in the request, so we can check that in the if statement that was added in HADOOP-8855 .
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12548267/HADOOP-8883.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-auth.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1574//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1574//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12548267/HADOOP-8883.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-auth. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1574//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1574//console This message is automatically generated.
          Hide
          Robert Kanter added a comment -

          By the way, Jenkins didn't test it (because its not configured with Kerberos?), but I checked that all tests in org.apache.hadoop.security.authentication.client.TestKerberosAuthenticator pass.

          Show
          Robert Kanter added a comment - By the way, Jenkins didn't test it (because its not configured with Kerberos?), but I checked that all tests in org.apache.hadoop.security.authentication.client.TestKerberosAuthenticator pass.
          Hide
          Alejandro Abdelnur added a comment -

          +1

          Show
          Alejandro Abdelnur added a comment - +1
          Hide
          Alejandro Abdelnur added a comment -

          Thanks Robert. Committed to trunk and branch-2.

          Show
          Alejandro Abdelnur added a comment - Thanks Robert. Committed to trunk and branch-2.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-trunk-Commit #2871 (See https://builds.apache.org/job/Hadoop-trunk-Commit/2871/)
          HADOOP-8883. Anonymous fallback in KerberosAuthenticator is broken. (rkanter via tucu) (Revision 1398895)

          Result = SUCCESS
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1398895
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/client/KerberosAuthenticator.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/TestKerberosAuthenticator.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          Hudson added a comment - Integrated in Hadoop-trunk-Commit #2871 (See https://builds.apache.org/job/Hadoop-trunk-Commit/2871/ ) HADOOP-8883 . Anonymous fallback in KerberosAuthenticator is broken. (rkanter via tucu) (Revision 1398895) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1398895 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/client/KerberosAuthenticator.java /hadoop/common/trunk/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java /hadoop/common/trunk/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/TestKerberosAuthenticator.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Yarn-trunk #6 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/6/)
          HADOOP-8883. Anonymous fallback in KerberosAuthenticator is broken. (rkanter via tucu) (Revision 1398895)

          Result = FAILURE
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1398895
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/client/KerberosAuthenticator.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/TestKerberosAuthenticator.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          Hudson added a comment - Integrated in Hadoop-Yarn-trunk #6 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/6/ ) HADOOP-8883 . Anonymous fallback in KerberosAuthenticator is broken. (rkanter via tucu) (Revision 1398895) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1398895 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/client/KerberosAuthenticator.java /hadoop/common/trunk/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java /hadoop/common/trunk/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/TestKerberosAuthenticator.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1198 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1198/)
          HADOOP-8883. Anonymous fallback in KerberosAuthenticator is broken. (rkanter via tucu) (Revision 1398895)

          Result = SUCCESS
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1398895
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/client/KerberosAuthenticator.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/TestKerberosAuthenticator.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1198 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1198/ ) HADOOP-8883 . Anonymous fallback in KerberosAuthenticator is broken. (rkanter via tucu) (Revision 1398895) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1398895 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/client/KerberosAuthenticator.java /hadoop/common/trunk/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java /hadoop/common/trunk/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/TestKerberosAuthenticator.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1228 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1228/)
          HADOOP-8883. Anonymous fallback in KerberosAuthenticator is broken. (rkanter via tucu) (Revision 1398895)

          Result = FAILURE
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1398895
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/client/KerberosAuthenticator.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/TestKerberosAuthenticator.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1228 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1228/ ) HADOOP-8883 . Anonymous fallback in KerberosAuthenticator is broken. (rkanter via tucu) (Revision 1398895) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1398895 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/client/KerberosAuthenticator.java /hadoop/common/trunk/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java /hadoop/common/trunk/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/TestKerberosAuthenticator.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          Andrey Klochkov added a comment -

          Actually JDK does not allow to read the content of "Authorization" request property so this fix doesn't change the behavior.

          Here's an extract from OpenJDK 1.7 sources. OpenJDK 1.6 is similar. My experiments with Oracle JDK7 shows the same behavior (the property is not available to the user).

                249     // the following http request headers should NOT have their values
                250     // returned for security reasons.
                251     private static final String[] EXCLUDE_HEADERS = {
                252             "Proxy-Authorization",
                253             "Authorization"
                254     };
          
          
               2709     @Override
               2710     public synchronized String getRequestProperty (String key) {
               2711         if (key == null) {
               2712             return null;
               2713         }
               2714 
               2715         // don't return headers containing security sensitive information
               2716         for (int i=0; i < EXCLUDE_HEADERS.length; i++) {
               2717             if (key.equalsIgnoreCase(EXCLUDE_HEADERS[i])) {
               2718                 return null;
               2719             }
               2720         }
          

          Should this Jira be re-opened or another one created?

          Show
          Andrey Klochkov added a comment - Actually JDK does not allow to read the content of "Authorization" request property so this fix doesn't change the behavior. Here's an extract from OpenJDK 1.7 sources. OpenJDK 1.6 is similar. My experiments with Oracle JDK7 shows the same behavior (the property is not available to the user). 249 // the following http request headers should NOT have their values 250 // returned for security reasons. 251 private static final String [] EXCLUDE_HEADERS = { 252 "Proxy-Authorization" , 253 "Authorization" 254 }; 2709 @Override 2710 public synchronized String getRequestProperty ( String key) { 2711 if (key == null ) { 2712 return null ; 2713 } 2714 2715 // don't return headers containing security sensitive information 2716 for ( int i=0; i < EXCLUDE_HEADERS.length; i++) { 2717 if (key.equalsIgnoreCase(EXCLUDE_HEADERS[i])) { 2718 return null ; 2719 } 2720 } Should this Jira be re-opened or another one created?
          Hide
          Robert Kanter added a comment -

          Looking at that code snippet, it does indeed seem like that the Authorization header would be excluded and return null. But then it seems weird that this had fixed the problem. There's a unit test and we saw it fix the issue in OOZIE-1010.

          Is it possible that OpenJDK 1.7, OpenJDK 1.6, and Oracle JDK 7 exclude the header but Oracle JDK 6 does not? If so, then this could be a JDK compatibility issue, and we should create a new JIRA to figure out a new way of fixing this.

          Can you check if the unit test in the patch TestKerberosAuthenticator#testFallbacktoPseudoAuthenticatorAnonymous fails on OpenJDK 1.7, OpenJDK 1.6, or Oracle JDK 7? I'm sure it passes on Oracle JDK 6.

          Show
          Robert Kanter added a comment - Looking at that code snippet, it does indeed seem like that the Authorization header would be excluded and return null . But then it seems weird that this had fixed the problem. There's a unit test and we saw it fix the issue in OOZIE-1010 . Is it possible that OpenJDK 1.7, OpenJDK 1.6, and Oracle JDK 7 exclude the header but Oracle JDK 6 does not? If so, then this could be a JDK compatibility issue, and we should create a new JIRA to figure out a new way of fixing this. Can you check if the unit test in the patch TestKerberosAuthenticator#testFallbacktoPseudoAuthenticatorAnonymous fails on OpenJDK 1.7, OpenJDK 1.6, or Oracle JDK 7? I'm sure it passes on Oracle JDK 6.
          Hide
          Robert Kanter added a comment -

          Alejandro told me offline that he checked and Oracle JDK 1.6 does actually exclude the Authorization header as well. I know that the patch fixed the problem we were having, so now I'm confused as to why it was even working... I'll try to look into this.

          Show
          Robert Kanter added a comment - Alejandro told me offline that he checked and Oracle JDK 1.6 does actually exclude the Authorization header as well. I know that the patch fixed the problem we were having, so now I'm confused as to why it was even working... I'll try to look into this.
          Hide
          Andrey Klochkov added a comment -

          Robert, TestKerberosAuthenticator#testFallbacktoPseudoAuthenticatorAnonymous passes, but it doesn't verify the presence of the Authorization header, it verifies absence of it. Effectively the check "conn.getRequestProperty(AUTHORIZATION) != null" would never succeed, so the first block in the code below would never be used.

          if (conn.getRequestProperty(AUTHORIZATION) != null && conn.getResponseCode() == HttpURLConnection.HTTP_OK) {
                  LOG.debug("JDK performed authentication on our behalf.");
                  // If the JDK already did the SPNEGO back-and-forth for
                  // us, just pull out the token.
                  AuthenticatedURL.extractToken(conn, token);
                  return;
                } else if (isNegotiate()) {
                  LOG.debug("Performing our own SPNEGO sequence.");
                  doSpnegoSequence(token);
                } else {
                  LOG.debug("Using fallback authenticator sequence.");
                  getFallBackAuthenticator().authenticate(url, token);
                }
          

          Can you please give me more context on why it is needed to do "manual" authorization, i.e. why JDK does not do the authentication every time, i.e. in what cases KerberosAuthenticator.doSpnegoSequence should be used? I'm interested in this as we're fighting with a related issue - in some circumstances JDK just stops doing the authentication we rely on (it stops caching tokens), and so all requests associated with HttpOpParam.Op instances with {requreAuth==false}} start failing – effectively, it's all operations except GETDELEGATIONTOKEN. If o.a.h.hdfs.web.URLConnectionFactory.openConnection used KerberosAuthenticator for all operations (not just getting delegation tokens), then we wouldn't have the problem. So I'm wondering who can shed some light on how JDK is supposed to handle that, and in what cases KerberosAuthenticator.doSpnegoSequence is really needed. Sorry, it's not related directly to this issue, I'm just trying to find who can have answers.

          Show
          Andrey Klochkov added a comment - Robert, TestKerberosAuthenticator#testFallbacktoPseudoAuthenticatorAnonymous passes, but it doesn't verify the presence of the Authorization header, it verifies absence of it. Effectively the check "conn.getRequestProperty(AUTHORIZATION) != null" would never succeed, so the first block in the code below would never be used. if (conn.getRequestProperty(AUTHORIZATION) != null && conn.getResponseCode() == HttpURLConnection.HTTP_OK) { LOG.debug( "JDK performed authentication on our behalf." ); // If the JDK already did the SPNEGO back-and-forth for // us, just pull out the token. AuthenticatedURL.extractToken(conn, token); return ; } else if (isNegotiate()) { LOG.debug( "Performing our own SPNEGO sequence." ); doSpnegoSequence(token); } else { LOG.debug( "Using fallback authenticator sequence." ); getFallBackAuthenticator().authenticate(url, token); } Can you please give me more context on why it is needed to do "manual" authorization, i.e. why JDK does not do the authentication every time, i.e. in what cases KerberosAuthenticator.doSpnegoSequence should be used? I'm interested in this as we're fighting with a related issue - in some circumstances JDK just stops doing the authentication we rely on (it stops caching tokens), and so all requests associated with HttpOpParam.Op instances with {requreAuth==false}} start failing – effectively, it's all operations except GETDELEGATIONTOKEN . If o.a.h.hdfs.web.URLConnectionFactory.openConnection used KerberosAuthenticator for all operations (not just getting delegation tokens), then we wouldn't have the problem. So I'm wondering who can shed some light on how JDK is supposed to handle that, and in what cases KerberosAuthenticator.doSpnegoSequence is really needed. Sorry, it's not related directly to this issue, I'm just trying to find who can have answers.
          Hide
          Robert Kanter added a comment -

          I was talking to Alejandro Abdelnur about this, and he could go into more detail, but from what I understand, the JDK doesn't quite implement the spec correctly, and in some cases it will do SPNEGO when we weren't expecting it to yet. So, while we can do it again (as the code always does now), that's wasteful and we can just extract the token in that case (what the code is supposed to be doing).

          In any case, you are correct that the first if block is never executed because of the change introduced by this JIRA. I'll work on a fix for that and create a new JIRA soon.

          Show
          Robert Kanter added a comment - I was talking to Alejandro Abdelnur about this, and he could go into more detail, but from what I understand, the JDK doesn't quite implement the spec correctly, and in some cases it will do SPNEGO when we weren't expecting it to yet. So, while we can do it again (as the code always does now), that's wasteful and we can just extract the token in that case (what the code is supposed to be doing). In any case, you are correct that the first if block is never executed because of the change introduced by this JIRA. I'll work on a fix for that and create a new JIRA soon.
          Hide
          Robert Kanter added a comment -

          I created HADOOP-10078 to fix the if statement.

          Show
          Robert Kanter added a comment - I created HADOOP-10078 to fix the if statement.

            People

            • Assignee:
              Robert Kanter
              Reporter:
              Robert Kanter
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development