Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-7988

Upper case in hostname part of the principals doesn't work with kerberos.

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 1.0.0, 0.23.1
    • 1.0.1, 1.1.0, 0.23.1, 2.0.0-alpha
    • None
    • None

    Description

      Kerberos doesn't like upper case in the hostname part of the principals.
      This issue has been seen in 23 as well as 1.0.

      Attachments

        1. HADOOP-7988.branch-1.patch
          2 kB
          Jitendra Nath Pandey
        2. HADOOP-7988.branch-1.patch
          0.6 kB
          Jitendra Nath Pandey
        3. HADOOP-7988.trunk.patch
          2 kB
          Jitendra Nath Pandey

        Issue Links

          Activity

            Kerberos is case sensitive. So if the local hostname returns upper case, this patch will likely break that machine from working.

            aw Allen Wittenauer added a comment - Kerberos is case sensitive. So if the local hostname returns upper case, this patch will likely break that machine from working.

            It is expected that principals will be created with lower case in the hostname in the kdc. The patch makes sure, that lower case hostname is used in the principal. The patch has been tested on machines having upper case in their hostnames.

            jnp Jitendra Nath Pandey added a comment - It is expected that principals will be created with lower case in the hostname in the kdc. The patch makes sure, that lower case hostname is used in the principal. The patch has been tested on machines having upper case in their hostnames.

            I'm -1 this patch without a config option to turn this behavior off.

            aw Allen Wittenauer added a comment - I'm -1 this patch without a config option to turn this behavior off.

            Oh, and before I forget, this should not be the default behavior.

            aw Allen Wittenauer added a comment - Oh, and before I forget, this should not be the default behavior.

            The issue this jira is trying to fix exists in both 0.23 and branch-1.

            Following excpetion was observed in 0.23, when we have upper case in the principals.

            2012-01-20 12:03:08,258 INFO ipc.HadoopYarnRPC (HadoopYarnProtoRPC.java:getProxy(48)) - Creating a HadoopYarnProtoRpc proxy for protocol interface org.apache.hadoop.yarn.server.api.ResourceTracker
            2012-01-20 12:03:08,291 INFO nodemanager.NodeStatusUpdaterImpl (NodeStatusUpdaterImpl.java:registerWithRM(155)) - Connected to ResourceManager at hadoopRM.example.aurora:9003
            2012-01-20 12:03:20,399 WARN ipc.Client (Client.java:run(526)) - Couldn't setup connection fornm/hadoopNM.example.aurora@EXAMPLE.AURORA to rm/hadoopRM.example.aurora@EXAMPLE.AURORA
            2012-01-20 12:03:20,405 ERROR service.CompositeService (CompositeService.java:start(72)) - Error starting services org.apache.hadoop.yarn.server.nodemanager.NodeManager
            org.apache.avro.AvroRuntimeException: java.lang.reflect.UndeclaredThrowableException
            at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.start(NodeStatusUpdaterImpl.java:132)
            at org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68)
            at org.apache.hadoop.yarn.server.nodemanager.NodeManager.start(NodeManager.java:163)
            at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:231)
            Caused by: java.lang.reflect.UndeclaredThrowableException
            at org.apache.hadoop.yarn.server.api.impl.pb.client.ResourceTrackerPBClientImpl.registerNodeManager(ResourceTrackerPBClientImpl.java:66)
            at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.registerWithRM(NodeStatusUpdaterImpl.java:161)
            at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.start(NodeStatusUpdaterImpl.java:128)
            ... 3 more
            Caused by: com.google.protobuf.ServiceException: java.io.IOException: Failed on local exception: java.io.IOException: Couldn't setup connection for nm/hadoopNM.example.aurora@EXAMPLE.AURORA to rm/hadoopRM.example.aurora@EXAMPLE.AURORA; Host Details : local host is: "hadoopNM/10.112.127.102"; destination host is: ""hadoopRM.example.aurora":9003;
            at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:139)
            at $Proxy14.registerNodeManager(Unknown Source)
            at org.apache.hadoop.yarn.server.api.impl.pb.client.ResourceTrackerPBClientImpl.registerNodeManager(ResourceTrackerPBClientImpl.java:59)
            ... 5 more
            Caused by: java.io.IOException: Failed on local exception: java.io.IOException: Couldn't setup connection fornm/hadoopNM.example.aurora@EXAMPLE.AURORA to rm/hadoopRM.example.aurora@EXAMPLE.AURORA; Host Details : local host is: "hadoopNM/10.112.127.102"; destination host is: ""hadoopRM.example.aurora":9003;
            at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:655)
            at org.apache.hadoop.ipc.Client.call(Client.java:1089)
            at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:136)
            ... 7 more
            Caused by: java.io.IOException: Couldn't setup connection for nm/hadoopNM.example.aurora@EXAMPLE.AURORA torm/hadoopRM.example.aurora@EXAMPLE.AURORA
            at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:527)
            at java.security.AccessController.doPrivileged(Native Method)
            at javax.security.auth.Subject.doAs(Subject.java:396)
            at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1152)
            at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:499)
            at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:583)
            at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:205)
            at org.apache.hadoop.ipc.Client.getConnection(Client.java:1195)
            at org.apache.hadoop.ipc.Client.call(Client.java:1065)
            ... 8 more
            Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - UNKNOWN_SERVER)]
            at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:194)
            at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:137)
            at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:407)
            at org.apache.hadoop.ipc.Client$Connection.access$1200(Client.java:205)
            at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:576)
            at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:573)
            at java.security.AccessController.doPrivileged(Native Method)
            at javax.security.auth.Subject.doAs(Subject.java:396)
            at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1152)
            at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:572)
            ... 11 more
            Caused by: GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - UNKNOWN_SERVER)
            at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:663)
            at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:230)
            at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:162)
            at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:175)
            ... 20 more
            Caused by: KrbException: Server not found in Kerberos database (7) - UNKNOWN_SERVER
            at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:64)
            at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:185)
            at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:294)
            at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:106)
            at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:557)
            at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:594)
            ... 23 more
            Caused by: KrbException: Identifier doesn't match expected value (906)
            at sun.security.krb5.internal.KDCRep.init(KDCRep.java:133)
            at sun.security.krb5.internal.TGSRep.init(TGSRep.java:58)
            at sun.security.krb5.internal.TGSRep.<init>(TGSRep.java:53)
            at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:46)
            ... 28 more

            jnp Jitendra Nath Pandey added a comment - The issue this jira is trying to fix exists in both 0.23 and branch-1. Following excpetion was observed in 0.23, when we have upper case in the principals. 2012-01-20 12:03:08,258 INFO ipc.HadoopYarnRPC (HadoopYarnProtoRPC.java:getProxy(48)) - Creating a HadoopYarnProtoRpc proxy for protocol interface org.apache.hadoop.yarn.server.api.ResourceTracker 2012-01-20 12:03:08,291 INFO nodemanager.NodeStatusUpdaterImpl (NodeStatusUpdaterImpl.java:registerWithRM(155)) - Connected to ResourceManager at hadoopRM.example.aurora:9003 2012-01-20 12:03:20,399 WARN ipc.Client (Client.java:run(526)) - Couldn't setup connection fornm/hadoopNM.example.aurora@EXAMPLE.AURORA to rm/hadoopRM.example.aurora@EXAMPLE.AURORA 2012-01-20 12:03:20,405 ERROR service.CompositeService (CompositeService.java:start(72)) - Error starting services org.apache.hadoop.yarn.server.nodemanager.NodeManager org.apache.avro.AvroRuntimeException: java.lang.reflect.UndeclaredThrowableException at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.start(NodeStatusUpdaterImpl.java:132) at org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.start(NodeManager.java:163) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:231) Caused by: java.lang.reflect.UndeclaredThrowableException at org.apache.hadoop.yarn.server.api.impl.pb.client.ResourceTrackerPBClientImpl.registerNodeManager(ResourceTrackerPBClientImpl.java:66) at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.registerWithRM(NodeStatusUpdaterImpl.java:161) at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.start(NodeStatusUpdaterImpl.java:128) ... 3 more Caused by: com.google.protobuf.ServiceException: java.io.IOException: Failed on local exception: java.io.IOException: Couldn't setup connection for nm/hadoopNM.example.aurora@EXAMPLE.AURORA to rm/hadoopRM.example.aurora@EXAMPLE.AURORA; Host Details : local host is: "hadoopNM/10.112.127.102"; destination host is: ""hadoopRM.example.aurora":9003; at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:139) at $Proxy14.registerNodeManager(Unknown Source) at org.apache.hadoop.yarn.server.api.impl.pb.client.ResourceTrackerPBClientImpl.registerNodeManager(ResourceTrackerPBClientImpl.java:59) ... 5 more Caused by: java.io.IOException: Failed on local exception: java.io.IOException: Couldn't setup connection fornm/hadoopNM.example.aurora@EXAMPLE.AURORA to rm/hadoopRM.example.aurora@EXAMPLE.AURORA; Host Details : local host is: "hadoopNM/10.112.127.102"; destination host is: ""hadoopRM.example.aurora":9003; at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:655) at org.apache.hadoop.ipc.Client.call(Client.java:1089) at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:136) ... 7 more Caused by: java.io.IOException: Couldn't setup connection for nm/hadoopNM.example.aurora@EXAMPLE.AURORA torm/hadoopRM.example.aurora@EXAMPLE.AURORA at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:527) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1152) at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:499) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:583) at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:205) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1195) at org.apache.hadoop.ipc.Client.call(Client.java:1065) ... 8 more Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - UNKNOWN_SERVER)] at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:194) at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:137) at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:407) at org.apache.hadoop.ipc.Client$Connection.access$1200(Client.java:205) at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:576) at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:573) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1152) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:572) ... 11 more Caused by: GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - UNKNOWN_SERVER) at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:663) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:230) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:162) at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:175) ... 20 more Caused by: KrbException: Server not found in Kerberos database (7) - UNKNOWN_SERVER at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:64) at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:185) at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:294) at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:106) at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:557) at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:594) ... 23 more Caused by: KrbException: Identifier doesn't match expected value (906) at sun.security.krb5.internal.KDCRep.init(KDCRep.java:133) at sun.security.krb5.internal.TGSRep.init(TGSRep.java:58) at sun.security.krb5.internal.TGSRep.<init>(TGSRep.java:53) at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:46) ... 28 more

            The patch only tries to standardize that hadoop always uses lowercase in the hostname part of the kerberos principal, making sure that kerberos always sees lower case in the hostname part of the principals. The patch doesn't change the hostname anywhere, only the principals used are affected.

            Just for reference of a similar code, hbase code uses same SecurityUtil api and explicitly converts the hostname to lowercase.

            jnp Jitendra Nath Pandey added a comment - The patch only tries to standardize that hadoop always uses lowercase in the hostname part of the kerberos principal, making sure that kerberos always sees lower case in the hostname part of the principals. The patch doesn't change the hostname anywhere, only the principals used are affected. Just for reference of a similar code, hbase code uses same SecurityUtil api and explicitly converts the hostname to lowercase.

            Ask me if I care what HBase does.

            However, I did have a chat with a friend of mine who reminded me that RFC 4150 (which I completely forgot about) actually does clarify the previous, murky rules around server principals by saying that the host component for case insensitive systems should be forced lower case. Note that this is for server principals. We are back in slightly murky waters again when you consider KSSL and SPNEGO....

            But even so, I'll remove my -1.

            aw Allen Wittenauer added a comment - Ask me if I care what HBase does. However, I did have a chat with a friend of mine who reminded me that RFC 4150 (which I completely forgot about) actually does clarify the previous, murky rules around server principals by saying that the host component for case insensitive systems should be forced lower case. Note that this is for server principals. We are back in slightly murky waters again when you consider KSSL and SPNEGO.... But even so, I'll remove my -1.
            mattf Matthew Foley added a comment -

            This is a complex topic. I'll try to summarize what I've found, but I'll also give the full reference hyperlinks.

            The Kerberos RFC says this:

            6.2.1. Name of Server Principals

            ... The first component of the
            two- or multi-component name will identify the service, and the
            latter components will identify the host. Where the name of the host
            is not case sensitive (for example, with Internet domain names) the
            name of the host MUST be lowercase.

            The Oracle Java Docs say this: Kerberos Requirements

            By convention, all Kerberos realm names are uppercase and all DNS hostname and domain names are lowercase. On the Windows 2000 platform, domains are also Kerberos realms; however, the realm name is always the uppercase version of the domain name.

            Hostnames are case insensitive and by convention they are all lowercase. They must resolve to the same hostname on the client and server by their respective naming services.

            However, in the Kerberos database hostnames are case sensitive. In all host-based Kerberos service principals in the KDC, hostnames are case-sensitive. The hostnames used in the Kerberos service principal names must exactly match the hostnames returned by the naming service. For example, if the naming service returns a fully qualified lowercased DNS hostname, such as "raven.sun.com", then the administrator must use the same fully qualified lowercased DNS hostname when creating host-based principal names in the KDC: "host/raven.sun.com".

            Finally, in an older IETF Draft related to Distributing Kerberos KDC and Realm Information with DNS the following commentary is provided:

            DNS vs. Kerberos - Case Sensitivity of Realm Names

            In Kerberos, realm names are case sensitive. While it is strongly
            encouraged that all realm names be all upper case this recommendation
            has not been adopted by all sites. Some sites use all lower case
            names and other use mixed case. DNS on the other hand is case insen-
            sitive for queries but is case preserving for responses to TXT
            queries. Since "MYREALM", "myrealm", and "MyRealm" are all different
            it is necessary that only one of the possible combinations of upper
            and lower case characters be used. This restriction may be lifted in
            the future as the DNS naming scheme is expanded to support non-ASCII
            names.

            These are the only authoritative references I could find. However, there is a considerable body of archived email list correspondence that strongly implies the Java Kerberos packages work best if host names are forced to lower case in the Kerberos principle names.

            The key issue seems to be the statement in the second reference, "The hostnames used in the Kerberos service principal names must exactly match the hostnames returned by the naming service," plus the observation in the third reference, {{ "it is necessary that only one of the possible combinations of upper and lower case characters be used [the same way in Kerberos and DNS]." }}

            So, if DNS servers and Kerberos servers disagree on hostname case usage and/or sensitivity, authentication won't work. For example, see this detailed complaint from 2010, mixed case hostname issue , where the response from a Kerberos contributor was simply "Hostnames are always case folded (to lower case) in principal names."

            There are also threads that imply the Java Kerberos packages may have or had bugs around this issue that are resolved by forcing hostnames to lower case. For example, see
            http://www.nexentastor.org/projects/site/wiki/CIFS :

            There is a known bug in the kerberos package which breaks ticketing in Nexenta if your domain name (in the case of active directory users) is mixed case or all upper case. Because kerberos requires all lower case this causes ticket errors that can be hard to trace down.


            and http://linux-nfs.org/pipermail/nfsv4/2005-July/002278.html :

            Browsing the code showed that handling keytab file has been done in
            different ways in Server and Client. While the server code mandates
            hostnames/principals to be of lowercase, client code doesn't impose any
            restriction...
            The current theory on the client was that we would use any usable
            key we found to do the mount. It can be argued that with
            authenticated delegation callback, the client becomes a server
            and therefore the server principal name restriction should apply.
            If everyone agrees with that, I will make the changes to restrict
            the principal name to be used on the client.

            To find other references, search the Web for "kerberos mixed case" and "kerberos requires lowercase".

            It seems to me that given the above, it makes sense that both HBase and Hadoop would choose to fold hostnames to lower case, in the Kerberos principals only.

            mattf Matthew Foley added a comment - This is a complex topic. I'll try to summarize what I've found, but I'll also give the full reference hyperlinks. The Kerberos RFC says this: 6.2.1. Name of Server Principals ... The first component of the two- or multi-component name will identify the service, and the latter components will identify the host. Where the name of the host is not case sensitive (for example, with Internet domain names) the name of the host MUST be lowercase. The Oracle Java Docs say this: Kerberos Requirements By convention, all Kerberos realm names are uppercase and all DNS hostname and domain names are lowercase. On the Windows 2000 platform, domains are also Kerberos realms; however, the realm name is always the uppercase version of the domain name. Hostnames are case insensitive and by convention they are all lowercase. They must resolve to the same hostname on the client and server by their respective naming services. However, in the Kerberos database hostnames are case sensitive. In all host-based Kerberos service principals in the KDC, hostnames are case-sensitive. The hostnames used in the Kerberos service principal names must exactly match the hostnames returned by the naming service. For example, if the naming service returns a fully qualified lowercased DNS hostname, such as "raven.sun.com", then the administrator must use the same fully qualified lowercased DNS hostname when creating host-based principal names in the KDC: "host/raven.sun.com". Finally, in an older IETF Draft related to Distributing Kerberos KDC and Realm Information with DNS the following commentary is provided: DNS vs. Kerberos - Case Sensitivity of Realm Names In Kerberos, realm names are case sensitive. While it is strongly encouraged that all realm names be all upper case this recommendation has not been adopted by all sites. Some sites use all lower case names and other use mixed case. DNS on the other hand is case insen- sitive for queries but is case preserving for responses to TXT queries. Since "MYREALM", "myrealm", and "MyRealm" are all different it is necessary that only one of the possible combinations of upper and lower case characters be used. This restriction may be lifted in the future as the DNS naming scheme is expanded to support non-ASCII names. These are the only authoritative references I could find. However, there is a considerable body of archived email list correspondence that strongly implies the Java Kerberos packages work best if host names are forced to lower case in the Kerberos principle names. The key issue seems to be the statement in the second reference, "The hostnames used in the Kerberos service principal names must exactly match the hostnames returned by the naming service," plus the observation in the third reference, {{ "it is necessary that only one of the possible combinations of upper and lower case characters be used [the same way in Kerberos and DNS] ." }} So, if DNS servers and Kerberos servers disagree on hostname case usage and/or sensitivity, authentication won't work. For example, see this detailed complaint from 2010, mixed case hostname issue , where the response from a Kerberos contributor was simply "Hostnames are always case folded (to lower case) in principal names." There are also threads that imply the Java Kerberos packages may have or had bugs around this issue that are resolved by forcing hostnames to lower case. For example, see http://www.nexentastor.org/projects/site/wiki/CIFS : There is a known bug in the kerberos package which breaks ticketing in Nexenta if your domain name (in the case of active directory users) is mixed case or all upper case. Because kerberos requires all lower case this causes ticket errors that can be hard to trace down. and http://linux-nfs.org/pipermail/nfsv4/2005-July/002278.html : Browsing the code showed that handling keytab file has been done in different ways in Server and Client. While the server code mandates hostnames/principals to be of lowercase, client code doesn't impose any restriction... The current theory on the client was that we would use any usable key we found to do the mount. It can be argued that with authenticated delegation callback, the client becomes a server and therefore the server principal name restriction should apply. If everyone agrees with that, I will make the changes to restrict the principal name to be used on the client. To find other references, search the Web for "kerberos mixed case" and "kerberos requires lowercase". It seems to me that given the above, it makes sense that both HBase and Hadoop would choose to fold hostnames to lower case, in the Kerberos principals only.

            So, if DNS servers and Kerberos servers disagree on hostname case usage and/or sensitivity, authentication won't work.

            Right. Which is why prior to 4150, forcing case at the app layer could result in all sorts of surprising behavior since, as you pointed out, it was ill-defined. The expectation was that the Kerberos principals would be set up to match DNS in those cases where DNS itself didn't return lower case. So if your DNS server responded with hostname.Example.ORG, then your host principal would be host/hostname.Example.ORG. From an administrative perspective, this was worked around by including both the "real" name and the folded name principals in the realm and associated keytabs.

            aw Allen Wittenauer added a comment - So, if DNS servers and Kerberos servers disagree on hostname case usage and/or sensitivity, authentication won't work. Right. Which is why prior to 4150, forcing case at the app layer could result in all sorts of surprising behavior since, as you pointed out, it was ill-defined. The expectation was that the Kerberos principals would be set up to match DNS in those cases where DNS itself didn't return lower case. So if your DNS server responded with hostname.Example.ORG, then your host principal would be host/hostname.Example.ORG. From an administrative perspective, this was worked around by including both the "real" name and the folded name principals in the realm and associated keytabs.
            mattf Matthew Foley added a comment - - edited

            Thanks, @Allen.

            @Jitendra, please provide a unit test, then provide patch for trunk and get auto-test to run.

            Thanks.

            mattf Matthew Foley added a comment - - edited Thanks, @Allen. @Jitendra, please provide a unit test, then provide patch for trunk and get auto-test to run. Thanks.

            Attached patch with unit tests.

            jnp Jitendra Nath Pandey added a comment - Attached patch with unit tests.

            Patch for trunk.

            jnp Jitendra Nath Pandey added a comment - Patch for trunk.
            hadoopqa Hadoop QA added a comment -

            +1 overall. Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12512250/HADOOP-7988.trunk.patch
            against trunk revision .

            +1 @author. The patch does not contain any @author tags.

            +1 tests included. The patch appears to include 3 new or modified tests.

            +1 javadoc. The javadoc tool did not generate any warning messages.

            +1 javac. The applied patch does not increase the total number of javac compiler warnings.

            +1 eclipse:eclipse. The patch built with eclipse:eclipse.

            +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

            +1 release audit. The applied patch does not increase the total number of release audit warnings.

            +1 core tests. The patch passed unit tests in .

            +1 contrib tests. The patch passed contrib unit tests.

            Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/544//testReport/
            Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/544//console

            This message is automatically generated.

            hadoopqa Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12512250/HADOOP-7988.trunk.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/544//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/544//console This message is automatically generated.
            mattf Matthew Foley added a comment -

            +1 for code review, both trunk and branch-1 versions. Thanks, Jitendra!

            mattf Matthew Foley added a comment - +1 for code review, both trunk and branch-1 versions. Thanks, Jitendra!
            hudson Hudson added a comment -

            Integrated in Hadoop-Hdfs-trunk-Commit #1683 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1683/)
            HADOOP-7988. Upper case in hostname part of the principals doesn't work with kerberos.

            jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236960
            Files :

            • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
            • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
            • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSecurityUtil.java
            hudson Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1683 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1683/ ) HADOOP-7988 . Upper case in hostname part of the principals doesn't work with kerberos. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236960 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSecurityUtil.java
            hudson Hudson added a comment -

            Integrated in Hadoop-Common-trunk-Commit #1611 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1611/)
            HADOOP-7988. Upper case in hostname part of the principals doesn't work with kerberos.

            jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236960
            Files :

            • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
            • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
            • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSecurityUtil.java
            hudson Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1611 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1611/ ) HADOOP-7988 . Upper case in hostname part of the principals doesn't work with kerberos. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236960 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSecurityUtil.java
            hudson Hudson added a comment -

            Integrated in Hadoop-Hdfs-0.23-Commit #432 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/432/)
            Merged r1236960 from trunk for HADOOP-7988.

            jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236962
            Files :

            • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
            • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
            • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSecurityUtil.java
            hudson Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #432 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/432/ ) Merged r1236960 from trunk for HADOOP-7988 . jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236962 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSecurityUtil.java
            hudson Hudson added a comment -

            Integrated in Hadoop-Common-0.23-Commit #441 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/441/)
            Merged r1236960 from trunk for HADOOP-7988.

            jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236962
            Files :

            • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
            • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
            • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSecurityUtil.java
            hudson Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #441 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/441/ ) Merged r1236960 from trunk for HADOOP-7988 . jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236962 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSecurityUtil.java
            hudson Hudson added a comment -

            Integrated in Hadoop-Mapreduce-trunk-Commit #1627 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1627/)
            HADOOP-7988. Upper case in hostname part of the principals doesn't work with kerberos.

            jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236960
            Files :

            • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
            • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
            • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSecurityUtil.java
            hudson Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1627 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1627/ ) HADOOP-7988 . Upper case in hostname part of the principals doesn't work with kerberos. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236960 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSecurityUtil.java
            hudson Hudson added a comment -

            Integrated in Hadoop-Mapreduce-0.23-Commit #456 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/456/)
            Merged r1236960 from trunk for HADOOP-7988.

            jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236962
            Files :

            • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
            • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
            • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSecurityUtil.java
            hudson Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #456 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/456/ ) Merged r1236960 from trunk for HADOOP-7988 . jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236962 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSecurityUtil.java
            hudson Hudson added a comment -

            Integrated in Hadoop-Hdfs-trunk #939 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/939/)
            HADOOP-7988. Upper case in hostname part of the principals doesn't work with kerberos.

            jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236960
            Files :

            • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
            • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
            • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSecurityUtil.java
            hudson Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #939 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/939/ ) HADOOP-7988 . Upper case in hostname part of the principals doesn't work with kerberos. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236960 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSecurityUtil.java
            hudson Hudson added a comment -

            Integrated in Hadoop-Mapreduce-0.23-Build #174 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/174/)
            Merged r1236960 from trunk for HADOOP-7988.

            jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236962
            Files :

            • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
            • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
            • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSecurityUtil.java
            hudson Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #174 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/174/ ) Merged r1236960 from trunk for HADOOP-7988 . jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236962 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSecurityUtil.java
            hudson Hudson added a comment -

            Integrated in Hadoop-Hdfs-0.23-Build #152 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/152/)
            Merged r1236960 from trunk for HADOOP-7988.

            jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236962
            Files :

            • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
            • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
            • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSecurityUtil.java
            hudson Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #152 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/152/ ) Merged r1236960 from trunk for HADOOP-7988 . jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236962 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSecurityUtil.java
            hudson Hudson added a comment -

            Integrated in Hadoop-Mapreduce-trunk #972 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/972/)
            HADOOP-7988. Upper case in hostname part of the principals doesn't work with kerberos.

            jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236960
            Files :

            • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
            • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
            • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSecurityUtil.java
            hudson Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #972 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/972/ ) HADOOP-7988 . Upper case in hostname part of the principals doesn't work with kerberos. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236960 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSecurityUtil.java
            mattf Matthew Foley added a comment -

            Closed upon release 1.0.1.

            mattf Matthew Foley added a comment - Closed upon release 1.0.1.

            People

              jnp Jitendra Nath Pandey
              jnp Jitendra Nath Pandey
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: