Hadoop Common
  1. Hadoop Common
  2. HADOOP-6632

Support for using different Kerberos keys for different instances of Hadoop services

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.22.0
    • Component/s: None
    • Labels:
      None

      Description

      We tested using the same Kerberos key for all datanodes in a HDFS cluster or the same Kerberos key for all TaskTarckers in a MapRed cluster. But it doesn't work. The reason is that when datanodes try to authenticate to the namenode all at once, the Kerberos authenticators they send to the namenode may have the same timestamp and will be rejected as replay requests. This JIRA makes it possible to use a unique key for each service instance.

      1. c6632-07.patch
        19 kB
        Kan Zhang
      2. c6632-05.patch
        19 kB
        Kan Zhang
      3. 6632.mr.patch
        2 kB
        Devaraj Das
      4. HADOOP-6632-Y20S-22.patch
        47 kB
        Jitendra Nath Pandey
      5. HADOOP-6632-Y20S-18.patch
        40 kB
        Kan Zhang

        Issue Links

          Activity

          Hide
          Kan Zhang added a comment -

          One error message we observed.

          2010-03-03 07:33:50,542 INFO org.apache.hadoop.ipc.Server: IPC Server listener on
          8020: readAndProcess threw exception javax.security.sasl.SaslException: GSS initia
          te failed [Caused by GSSException: Failure unspecified at GSS-API level (Mechanism
          level: Request is a replay (34))]. Count of bytes read: 0
          javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: Failure unspecified at GSS-API level
          (Mechanism level: Request is a replay (34))]
          at com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:159)
          at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:913)
          at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:1071)
          at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:459)
          at org.apache.hadoop.ipc.Server$Listener.run(Server.java:368)
          Caused by: GSSException: Failure unspecified at GSS-API level (Mechanism level: Request is a replay (34))
          at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:741)
          at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:323)
          at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:267)
          at com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:137)
          ... 4 more
          Caused by: KrbException: Request is a replay (34)
          at sun.security.krb5.KrbApReq.authenticate(KrbApReq.java:299)
          at sun.security.krb5.KrbApReq.<init>(KrbApReq.java:134)
          at sun.security.jgss.krb5.InitSecContextToken.<init>(InitSecContextToken.java:79)
          at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:724)
          ... 7 more

          Show
          Kan Zhang added a comment - One error message we observed. 2010-03-03 07:33:50,542 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 8020: readAndProcess threw exception javax.security.sasl.SaslException: GSS initia te failed [Caused by GSSException: Failure unspecified at GSS-API level (Mechanism level: Request is a replay (34))]. Count of bytes read: 0 javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: Failure unspecified at GSS-API level (Mechanism level: Request is a replay (34))] at com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:159) at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:913) at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:1071) at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:459) at org.apache.hadoop.ipc.Server$Listener.run(Server.java:368) Caused by: GSSException: Failure unspecified at GSS-API level (Mechanism level: Request is a replay (34)) at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:741) at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:323) at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:267) at com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:137) ... 4 more Caused by: KrbException: Request is a replay (34) at sun.security.krb5.KrbApReq.authenticate(KrbApReq.java:299) at sun.security.krb5.KrbApReq.<init>(KrbApReq.java:134) at sun.security.jgss.krb5.InitSecContextToken.<init>(InitSecContextToken.java:79) at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:724) ... 7 more
          Hide
          Kan Zhang added a comment -

          A patch for Yahoo 0.20S branch. Not for commit.

          This patch assumes a service's Kerberos principal name is of the form "servicename/hostname@REALM", where servicename is the service type (for example, dn for DataNodes, and tt for TaskTrackers), hostname is the fully-qualified domain name of the host where the service is running on, and REALM is Kerberos realm that the service belongs to. To support the convenience of having the same conf on every host, a service's Kerberos principal name can be configured as "servicename/$

          {FQDN}@REALM", where ${FQDN}

          will be substituted at runtime with the fully-qualified domain name of the host that the service is running on.

          Show
          Kan Zhang added a comment - A patch for Yahoo 0.20S branch. Not for commit. This patch assumes a service's Kerberos principal name is of the form "servicename/hostname@REALM", where servicename is the service type (for example, dn for DataNodes, and tt for TaskTrackers), hostname is the fully-qualified domain name of the host where the service is running on, and REALM is Kerberos realm that the service belongs to. To support the convenience of having the same conf on every host, a service's Kerberos principal name can be configured as "servicename/$ {FQDN}@REALM", where ${FQDN} will be substituted at runtime with the fully-qualified domain name of the host that the service is running on.
          Hide
          Jitendra Nath Pandey added a comment -

          New patch for hadoop-20.
          The hostname patter is changed to _HOST, and the renewer for delegation tokens is changed to shortnames.

          Show
          Jitendra Nath Pandey added a comment - New patch for hadoop-20. The hostname patter is changed to _HOST, and the renewer for delegation tokens is changed to shortnames.
          Hide
          Devaraj Das added a comment -

          A minor fix for the MR side to reuse filesystem handles

          Show
          Devaraj Das added a comment - A minor fix for the MR side to reuse filesystem handles
          Hide
          Kan Zhang added a comment -

          A port for trunk.

          Show
          Kan Zhang added a comment - A port for trunk.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12449571/c6632-05.patch
          against trunk revision 964134.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/615/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/615/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/615/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/615/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12449571/c6632-05.patch against trunk revision 964134. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/615/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/615/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/615/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/615/console This message is automatically generated.
          Hide
          Kan Zhang added a comment -

          The javadoc warnings are unrelated to this patch. Manually verified the feature on a single node cluster.

          Show
          Kan Zhang added a comment - The javadoc warnings are unrelated to this patch. Manually verified the feature on a single node cluster.
          Hide
          Kan Zhang added a comment -

          Uploading a new patch that simply merges with latest trunk changes. No semantic change from previous patch.

          Show
          Kan Zhang added a comment - Uploading a new patch that simply merges with latest trunk changes. No semantic change from previous patch.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12449900/c6632-07.patch
          against trunk revision 965556.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/624/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/624/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/624/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/624/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12449900/c6632-07.patch against trunk revision 965556. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/624/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/624/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/624/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/624/console This message is automatically generated.
          Hide
          Devaraj Das added a comment -

          I just committed this. Thanks, Kan & Jitendra!

          Show
          Devaraj Das added a comment - I just committed this. Thanks, Kan & Jitendra!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #331 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/331/)
          HADOOP-6632. Adds support for using different keytabs for different servers in a Hadoop cluster. In the earier implementation, all servers of a certain type (like TaskTracker), would have the same keytab and the same principal. Now the principal name is a pattern that has _HOST in it. Contributed by Kan Zhang & Jitendra Pandey.

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #331 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/331/ ) HADOOP-6632 . Adds support for using different keytabs for different servers in a Hadoop cluster. In the earier implementation, all servers of a certain type (like TaskTracker), would have the same keytab and the same principal. Now the principal name is a pattern that has _HOST in it. Contributed by Kan Zhang & Jitendra Pandey.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #346 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/346/)
          HDFS-1201. The HDFS component for HADOOP-6632. Contributed by Kan Zhang & Jitendra Pandey.

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #346 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/346/ ) HDFS-1201 . The HDFS component for HADOOP-6632 . Contributed by Kan Zhang & Jitendra Pandey.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk #398 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/398/)
          HADOOP-6632. Adds support for using different keytabs for different servers in a Hadoop cluster. In the earier implementation, all servers of a certain type (like TaskTracker), would have the same keytab and the same principal. Now the principal name is a pattern that has _HOST in it. Contributed by Kan Zhang & Jitendra Pandey.

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk #398 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/398/ ) HADOOP-6632 . Adds support for using different keytabs for different servers in a Hadoop cluster. In the earier implementation, all servers of a certain type (like TaskTracker), would have the same keytab and the same principal. Now the principal name is a pattern that has _HOST in it. Contributed by Kan Zhang & Jitendra Pandey.
          Hide
          Todd Lipcon added a comment -

          It looks like the 6632.mr.patch portion was committed to ydist but not trunk - was this intentional?

          Show
          Todd Lipcon added a comment - It looks like the 6632.mr.patch portion was committed to ydist but not trunk - was this intentional?
          Hide
          Devaraj Das added a comment -

          Yes this was intentional. The mr patch seemed like a hack and that's why we didn't commit it to trunk, and instead raised MAPREDUCE-1824 to discuss that... BTW, the problem which the mr patch attempted to address would be significantly less once we have HADOOP-6706 committed that does retries in case of failures due to the false replay attack detection by the rpc servers. MAPREDUCE-1824 takes a low priority..

          Show
          Devaraj Das added a comment - Yes this was intentional. The mr patch seemed like a hack and that's why we didn't commit it to trunk, and instead raised MAPREDUCE-1824 to discuss that... BTW, the problem which the mr patch attempted to address would be significantly less once we have HADOOP-6706 committed that does retries in case of failures due to the false replay attack detection by the rpc servers. MAPREDUCE-1824 takes a low priority..
          Hide
          Todd Lipcon added a comment -

          Thanks, Deveraj. That makes sense.

          Show
          Todd Lipcon added a comment - Thanks, Deveraj. That makes sense.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #523 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/523/)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #523 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/523/ )

            People

            • Assignee:
              Kan Zhang
              Reporter:
              Kan Zhang
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development