Hadoop Common
  1. Hadoop Common
  2. HADOOP-7988

Upper case in hostname part of the principals doesn't work with kerberos.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.0, 0.23.1
    • Fix Version/s: 1.0.1, 1.1.0, 0.23.1, 2.0.0-alpha
    • Component/s: None
    • Labels:
      None

      Description

      Kerberos doesn't like upper case in the hostname part of the principals.
      This issue has been seen in 23 as well as 1.0.

      1. HADOOP-7988.branch-1.patch
        0.6 kB
        Jitendra Nath Pandey
      2. HADOOP-7988.branch-1.patch
        2 kB
        Jitendra Nath Pandey
      3. HADOOP-7988.trunk.patch
        2 kB
        Jitendra Nath Pandey

        Issue Links

          Activity

          Hide
          Allen Wittenauer added a comment -

          Kerberos is case sensitive. So if the local hostname returns upper case, this patch will likely break that machine from working.

          Show
          Allen Wittenauer added a comment - Kerberos is case sensitive. So if the local hostname returns upper case, this patch will likely break that machine from working.
          Hide
          Jitendra Nath Pandey added a comment -

          It is expected that principals will be created with lower case in the hostname in the kdc. The patch makes sure, that lower case hostname is used in the principal. The patch has been tested on machines having upper case in their hostnames.

          Show
          Jitendra Nath Pandey added a comment - It is expected that principals will be created with lower case in the hostname in the kdc. The patch makes sure, that lower case hostname is used in the principal. The patch has been tested on machines having upper case in their hostnames.
          Hide
          Allen Wittenauer added a comment -

          I'm -1 this patch without a config option to turn this behavior off.

          Show
          Allen Wittenauer added a comment - I'm -1 this patch without a config option to turn this behavior off.
          Hide
          Allen Wittenauer added a comment -

          Oh, and before I forget, this should not be the default behavior.

          Show
          Allen Wittenauer added a comment - Oh, and before I forget, this should not be the default behavior.
          Hide
          Jitendra Nath Pandey added a comment -

          The issue this jira is trying to fix exists in both 0.23 and branch-1.

          Following excpetion was observed in 0.23, when we have upper case in the principals.

          2012-01-20 12:03:08,258 INFO ipc.HadoopYarnRPC (HadoopYarnProtoRPC.java:getProxy(48)) - Creating a HadoopYarnProtoRpc proxy for protocol interface org.apache.hadoop.yarn.server.api.ResourceTracker
          2012-01-20 12:03:08,291 INFO nodemanager.NodeStatusUpdaterImpl (NodeStatusUpdaterImpl.java:registerWithRM(155)) - Connected to ResourceManager at hadoopRM.example.aurora:9003
          2012-01-20 12:03:20,399 WARN ipc.Client (Client.java:run(526)) - Couldn't setup connection fornm/hadoopNM.example.aurora@EXAMPLE.AURORA to rm/hadoopRM.example.aurora@EXAMPLE.AURORA
          2012-01-20 12:03:20,405 ERROR service.CompositeService (CompositeService.java:start(72)) - Error starting services org.apache.hadoop.yarn.server.nodemanager.NodeManager
          org.apache.avro.AvroRuntimeException: java.lang.reflect.UndeclaredThrowableException
          at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.start(NodeStatusUpdaterImpl.java:132)
          at org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68)
          at org.apache.hadoop.yarn.server.nodemanager.NodeManager.start(NodeManager.java:163)
          at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:231)
          Caused by: java.lang.reflect.UndeclaredThrowableException
          at org.apache.hadoop.yarn.server.api.impl.pb.client.ResourceTrackerPBClientImpl.registerNodeManager(ResourceTrackerPBClientImpl.java:66)
          at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.registerWithRM(NodeStatusUpdaterImpl.java:161)
          at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.start(NodeStatusUpdaterImpl.java:128)
          ... 3 more
          Caused by: com.google.protobuf.ServiceException: java.io.IOException: Failed on local exception: java.io.IOException: Couldn't setup connection for nm/hadoopNM.example.aurora@EXAMPLE.AURORA to rm/hadoopRM.example.aurora@EXAMPLE.AURORA; Host Details : local host is: "hadoopNM/10.112.127.102"; destination host is: ""hadoopRM.example.aurora":9003;
          at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:139)
          at $Proxy14.registerNodeManager(Unknown Source)
          at org.apache.hadoop.yarn.server.api.impl.pb.client.ResourceTrackerPBClientImpl.registerNodeManager(ResourceTrackerPBClientImpl.java:59)
          ... 5 more
          Caused by: java.io.IOException: Failed on local exception: java.io.IOException: Couldn't setup connection fornm/hadoopNM.example.aurora@EXAMPLE.AURORA to rm/hadoopRM.example.aurora@EXAMPLE.AURORA; Host Details : local host is: "hadoopNM/10.112.127.102"; destination host is: ""hadoopRM.example.aurora":9003;
          at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:655)
          at org.apache.hadoop.ipc.Client.call(Client.java:1089)
          at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:136)
          ... 7 more
          Caused by: java.io.IOException: Couldn't setup connection for nm/hadoopNM.example.aurora@EXAMPLE.AURORA torm/hadoopRM.example.aurora@EXAMPLE.AURORA
          at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:527)
          at java.security.AccessController.doPrivileged(Native Method)
          at javax.security.auth.Subject.doAs(Subject.java:396)
          at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1152)
          at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:499)
          at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:583)
          at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:205)
          at org.apache.hadoop.ipc.Client.getConnection(Client.java:1195)
          at org.apache.hadoop.ipc.Client.call(Client.java:1065)
          ... 8 more
          Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - UNKNOWN_SERVER)]
          at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:194)
          at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:137)
          at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:407)
          at org.apache.hadoop.ipc.Client$Connection.access$1200(Client.java:205)
          at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:576)
          at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:573)
          at java.security.AccessController.doPrivileged(Native Method)
          at javax.security.auth.Subject.doAs(Subject.java:396)
          at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1152)
          at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:572)
          ... 11 more
          Caused by: GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - UNKNOWN_SERVER)
          at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:663)
          at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:230)
          at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:162)
          at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:175)
          ... 20 more
          Caused by: KrbException: Server not found in Kerberos database (7) - UNKNOWN_SERVER
          at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:64)
          at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:185)
          at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:294)
          at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:106)
          at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:557)
          at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:594)
          ... 23 more
          Caused by: KrbException: Identifier doesn't match expected value (906)
          at sun.security.krb5.internal.KDCRep.init(KDCRep.java:133)
          at sun.security.krb5.internal.TGSRep.init(TGSRep.java:58)
          at sun.security.krb5.internal.TGSRep.<init>(TGSRep.java:53)
          at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:46)
          ... 28 more

          Show
          Jitendra Nath Pandey added a comment - The issue this jira is trying to fix exists in both 0.23 and branch-1. Following excpetion was observed in 0.23, when we have upper case in the principals. 2012-01-20 12:03:08,258 INFO ipc.HadoopYarnRPC (HadoopYarnProtoRPC.java:getProxy(48)) - Creating a HadoopYarnProtoRpc proxy for protocol interface org.apache.hadoop.yarn.server.api.ResourceTracker 2012-01-20 12:03:08,291 INFO nodemanager.NodeStatusUpdaterImpl (NodeStatusUpdaterImpl.java:registerWithRM(155)) - Connected to ResourceManager at hadoopRM.example.aurora:9003 2012-01-20 12:03:20,399 WARN ipc.Client (Client.java:run(526)) - Couldn't setup connection fornm/hadoopNM.example.aurora@EXAMPLE.AURORA to rm/hadoopRM.example.aurora@EXAMPLE.AURORA 2012-01-20 12:03:20,405 ERROR service.CompositeService (CompositeService.java:start(72)) - Error starting services org.apache.hadoop.yarn.server.nodemanager.NodeManager org.apache.avro.AvroRuntimeException: java.lang.reflect.UndeclaredThrowableException at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.start(NodeStatusUpdaterImpl.java:132) at org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.start(NodeManager.java:163) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:231) Caused by: java.lang.reflect.UndeclaredThrowableException at org.apache.hadoop.yarn.server.api.impl.pb.client.ResourceTrackerPBClientImpl.registerNodeManager(ResourceTrackerPBClientImpl.java:66) at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.registerWithRM(NodeStatusUpdaterImpl.java:161) at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.start(NodeStatusUpdaterImpl.java:128) ... 3 more Caused by: com.google.protobuf.ServiceException: java.io.IOException: Failed on local exception: java.io.IOException: Couldn't setup connection for nm/hadoopNM.example.aurora@EXAMPLE.AURORA to rm/hadoopRM.example.aurora@EXAMPLE.AURORA; Host Details : local host is: "hadoopNM/10.112.127.102"; destination host is: ""hadoopRM.example.aurora":9003; at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:139) at $Proxy14.registerNodeManager(Unknown Source) at org.apache.hadoop.yarn.server.api.impl.pb.client.ResourceTrackerPBClientImpl.registerNodeManager(ResourceTrackerPBClientImpl.java:59) ... 5 more Caused by: java.io.IOException: Failed on local exception: java.io.IOException: Couldn't setup connection fornm/hadoopNM.example.aurora@EXAMPLE.AURORA to rm/hadoopRM.example.aurora@EXAMPLE.AURORA; Host Details : local host is: "hadoopNM/10.112.127.102"; destination host is: ""hadoopRM.example.aurora":9003; at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:655) at org.apache.hadoop.ipc.Client.call(Client.java:1089) at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:136) ... 7 more Caused by: java.io.IOException: Couldn't setup connection for nm/hadoopNM.example.aurora@EXAMPLE.AURORA torm/hadoopRM.example.aurora@EXAMPLE.AURORA at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:527) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1152) at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:499) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:583) at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:205) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1195) at org.apache.hadoop.ipc.Client.call(Client.java:1065) ... 8 more Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - UNKNOWN_SERVER)] at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:194) at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:137) at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:407) at org.apache.hadoop.ipc.Client$Connection.access$1200(Client.java:205) at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:576) at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:573) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1152) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:572) ... 11 more Caused by: GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - UNKNOWN_SERVER) at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:663) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:230) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:162) at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:175) ... 20 more Caused by: KrbException: Server not found in Kerberos database (7) - UNKNOWN_SERVER at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:64) at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:185) at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:294) at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:106) at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:557) at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:594) ... 23 more Caused by: KrbException: Identifier doesn't match expected value (906) at sun.security.krb5.internal.KDCRep.init(KDCRep.java:133) at sun.security.krb5.internal.TGSRep.init(TGSRep.java:58) at sun.security.krb5.internal.TGSRep.<init>(TGSRep.java:53) at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:46) ... 28 more
          Hide
          Jitendra Nath Pandey added a comment -

          The patch only tries to standardize that hadoop always uses lowercase in the hostname part of the kerberos principal, making sure that kerberos always sees lower case in the hostname part of the principals. The patch doesn't change the hostname anywhere, only the principals used are affected.

          Just for reference of a similar code, hbase code uses same SecurityUtil api and explicitly converts the hostname to lowercase.

          Show
          Jitendra Nath Pandey added a comment - The patch only tries to standardize that hadoop always uses lowercase in the hostname part of the kerberos principal, making sure that kerberos always sees lower case in the hostname part of the principals. The patch doesn't change the hostname anywhere, only the principals used are affected. Just for reference of a similar code, hbase code uses same SecurityUtil api and explicitly converts the hostname to lowercase.
          Hide
          Allen Wittenauer added a comment -

          Ask me if I care what HBase does.

          However, I did have a chat with a friend of mine who reminded me that RFC 4150 (which I completely forgot about) actually does clarify the previous, murky rules around server principals by saying that the host component for case insensitive systems should be forced lower case. Note that this is for server principals. We are back in slightly murky waters again when you consider KSSL and SPNEGO....

          But even so, I'll remove my -1.

          Show
          Allen Wittenauer added a comment - Ask me if I care what HBase does. However, I did have a chat with a friend of mine who reminded me that RFC 4150 (which I completely forgot about) actually does clarify the previous, murky rules around server principals by saying that the host component for case insensitive systems should be forced lower case. Note that this is for server principals. We are back in slightly murky waters again when you consider KSSL and SPNEGO.... But even so, I'll remove my -1.
          Hide
          Matt Foley added a comment -

          This is a complex topic. I'll try to summarize what I've found, but I'll also give the full reference hyperlinks.

          The Kerberos RFC says this:

          6.2.1. Name of Server Principals

          ... The first component of the
          two- or multi-component name will identify the service, and the
          latter components will identify the host. Where the name of the host
          is not case sensitive (for example, with Internet domain names) the
          name of the host MUST be lowercase.

          The Oracle Java Docs say this: Kerberos Requirements

          By convention, all Kerberos realm names are uppercase and all DNS hostname and domain names are lowercase. On the Windows 2000 platform, domains are also Kerberos realms; however, the realm name is always the uppercase version of the domain name.

          Hostnames are case insensitive and by convention they are all lowercase. They must resolve to the same hostname on the client and server by their respective naming services.

          However, in the Kerberos database hostnames are case sensitive. In all host-based Kerberos service principals in the KDC, hostnames are case-sensitive. The hostnames used in the Kerberos service principal names must exactly match the hostnames returned by the naming service. For example, if the naming service returns a fully qualified lowercased DNS hostname, such as "raven.sun.com", then the administrator must use the same fully qualified lowercased DNS hostname when creating host-based principal names in the KDC: "host/raven.sun.com".

          Finally, in an older IETF Draft related to Distributing Kerberos KDC and Realm Information with DNS the following commentary is provided:

          DNS vs. Kerberos - Case Sensitivity of Realm Names

          In Kerberos, realm names are case sensitive. While it is strongly
          encouraged that all realm names be all upper case this recommendation
          has not been adopted by all sites. Some sites use all lower case
          names and other use mixed case. DNS on the other hand is case insen-
          sitive for queries but is case preserving for responses to TXT
          queries. Since "MYREALM", "myrealm", and "MyRealm" are all different
          it is necessary that only one of the possible combinations of upper
          and lower case characters be used. This restriction may be lifted in
          the future as the DNS naming scheme is expanded to support non-ASCII
          names.

          These are the only authoritative references I could find. However, there is a considerable body of archived email list correspondence that strongly implies the Java Kerberos packages work best if host names are forced to lower case in the Kerberos principle names.

          The key issue seems to be the statement in the second reference, "The hostnames used in the Kerberos service principal names must exactly match the hostnames returned by the naming service," plus the observation in the third reference, {{ "it is necessary that only one of the possible combinations of upper and lower case characters be used [the same way in Kerberos and DNS]." }}

          So, if DNS servers and Kerberos servers disagree on hostname case usage and/or sensitivity, authentication won't work. For example, see this detailed complaint from 2010, mixed case hostname issue , where the response from a Kerberos contributor was simply "Hostnames are always case folded (to lower case) in principal names."

          There are also threads that imply the Java Kerberos packages may have or had bugs around this issue that are resolved by forcing hostnames to lower case. For example, see
          http://www.nexentastor.org/projects/site/wiki/CIFS :

          There is a known bug in the kerberos package which breaks ticketing in Nexenta if your domain name (in the case of active directory users) is mixed case or all upper case. Because kerberos requires all lower case this causes ticket errors that can be hard to trace down.


          and http://linux-nfs.org/pipermail/nfsv4/2005-July/002278.html :

          Browsing the code showed that handling keytab file has been done in
          different ways in Server and Client. While the server code mandates
          hostnames/principals to be of lowercase, client code doesn't impose any
          restriction...
          The current theory on the client was that we would use any usable
          key we found to do the mount. It can be argued that with
          authenticated delegation callback, the client becomes a server
          and therefore the server principal name restriction should apply.
          If everyone agrees with that, I will make the changes to restrict
          the principal name to be used on the client.

          To find other references, search the Web for "kerberos mixed case" and "kerberos requires lowercase".

          It seems to me that given the above, it makes sense that both HBase and Hadoop would choose to fold hostnames to lower case, in the Kerberos principals only.

          Show
          Matt Foley added a comment - This is a complex topic. I'll try to summarize what I've found, but I'll also give the full reference hyperlinks. The Kerberos RFC says this: 6.2.1. Name of Server Principals ... The first component of the two- or multi-component name will identify the service, and the latter components will identify the host. Where the name of the host is not case sensitive (for example, with Internet domain names) the name of the host MUST be lowercase. The Oracle Java Docs say this: Kerberos Requirements By convention, all Kerberos realm names are uppercase and all DNS hostname and domain names are lowercase. On the Windows 2000 platform, domains are also Kerberos realms; however, the realm name is always the uppercase version of the domain name. Hostnames are case insensitive and by convention they are all lowercase. They must resolve to the same hostname on the client and server by their respective naming services. However, in the Kerberos database hostnames are case sensitive. In all host-based Kerberos service principals in the KDC, hostnames are case-sensitive. The hostnames used in the Kerberos service principal names must exactly match the hostnames returned by the naming service. For example, if the naming service returns a fully qualified lowercased DNS hostname, such as "raven.sun.com", then the administrator must use the same fully qualified lowercased DNS hostname when creating host-based principal names in the KDC: "host/raven.sun.com". Finally, in an older IETF Draft related to Distributing Kerberos KDC and Realm Information with DNS the following commentary is provided: DNS vs. Kerberos - Case Sensitivity of Realm Names In Kerberos, realm names are case sensitive. While it is strongly encouraged that all realm names be all upper case this recommendation has not been adopted by all sites. Some sites use all lower case names and other use mixed case. DNS on the other hand is case insen- sitive for queries but is case preserving for responses to TXT queries. Since "MYREALM", "myrealm", and "MyRealm" are all different it is necessary that only one of the possible combinations of upper and lower case characters be used. This restriction may be lifted in the future as the DNS naming scheme is expanded to support non-ASCII names. These are the only authoritative references I could find. However, there is a considerable body of archived email list correspondence that strongly implies the Java Kerberos packages work best if host names are forced to lower case in the Kerberos principle names. The key issue seems to be the statement in the second reference, "The hostnames used in the Kerberos service principal names must exactly match the hostnames returned by the naming service," plus the observation in the third reference, {{ "it is necessary that only one of the possible combinations of upper and lower case characters be used [the same way in Kerberos and DNS] ." }} So, if DNS servers and Kerberos servers disagree on hostname case usage and/or sensitivity, authentication won't work. For example, see this detailed complaint from 2010, mixed case hostname issue , where the response from a Kerberos contributor was simply "Hostnames are always case folded (to lower case) in principal names." There are also threads that imply the Java Kerberos packages may have or had bugs around this issue that are resolved by forcing hostnames to lower case. For example, see http://www.nexentastor.org/projects/site/wiki/CIFS : There is a known bug in the kerberos package which breaks ticketing in Nexenta if your domain name (in the case of active directory users) is mixed case or all upper case. Because kerberos requires all lower case this causes ticket errors that can be hard to trace down. and http://linux-nfs.org/pipermail/nfsv4/2005-July/002278.html : Browsing the code showed that handling keytab file has been done in different ways in Server and Client. While the server code mandates hostnames/principals to be of lowercase, client code doesn't impose any restriction... The current theory on the client was that we would use any usable key we found to do the mount. It can be argued that with authenticated delegation callback, the client becomes a server and therefore the server principal name restriction should apply. If everyone agrees with that, I will make the changes to restrict the principal name to be used on the client. To find other references, search the Web for "kerberos mixed case" and "kerberos requires lowercase". It seems to me that given the above, it makes sense that both HBase and Hadoop would choose to fold hostnames to lower case, in the Kerberos principals only.
          Hide
          Allen Wittenauer added a comment -

          So, if DNS servers and Kerberos servers disagree on hostname case usage and/or sensitivity, authentication won't work.

          Right. Which is why prior to 4150, forcing case at the app layer could result in all sorts of surprising behavior since, as you pointed out, it was ill-defined. The expectation was that the Kerberos principals would be set up to match DNS in those cases where DNS itself didn't return lower case. So if your DNS server responded with hostname.Example.ORG, then your host principal would be host/hostname.Example.ORG. From an administrative perspective, this was worked around by including both the "real" name and the folded name principals in the realm and associated keytabs.

          Show
          Allen Wittenauer added a comment - So, if DNS servers and Kerberos servers disagree on hostname case usage and/or sensitivity, authentication won't work. Right. Which is why prior to 4150, forcing case at the app layer could result in all sorts of surprising behavior since, as you pointed out, it was ill-defined. The expectation was that the Kerberos principals would be set up to match DNS in those cases where DNS itself didn't return lower case. So if your DNS server responded with hostname.Example.ORG, then your host principal would be host/hostname.Example.ORG. From an administrative perspective, this was worked around by including both the "real" name and the folded name principals in the realm and associated keytabs.
          Hide
          Matt Foley added a comment - - edited

          Thanks, @Allen.

          @Jitendra, please provide a unit test, then provide patch for trunk and get auto-test to run.

          Thanks.

          Show
          Matt Foley added a comment - - edited Thanks, @Allen. @Jitendra, please provide a unit test, then provide patch for trunk and get auto-test to run. Thanks.
          Hide
          Jitendra Nath Pandey added a comment -

          Attached patch with unit tests.

          Show
          Jitendra Nath Pandey added a comment - Attached patch with unit tests.
          Hide
          Jitendra Nath Pandey added a comment -

          Patch for trunk.

          Show
          Jitendra Nath Pandey added a comment - Patch for trunk.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12512250/HADOOP-7988.trunk.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/544//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/544//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12512250/HADOOP-7988.trunk.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/544//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/544//console This message is automatically generated.
          Hide
          Matt Foley added a comment -

          +1 for code review, both trunk and branch-1 versions. Thanks, Jitendra!

          Show
          Matt Foley added a comment - +1 for code review, both trunk and branch-1 versions. Thanks, Jitendra!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #1683 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1683/)
          HADOOP-7988. Upper case in hostname part of the principals doesn't work with kerberos.

          jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236960
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSecurityUtil.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1683 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1683/ ) HADOOP-7988 . Upper case in hostname part of the principals doesn't work with kerberos. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236960 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSecurityUtil.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #1611 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1611/)
          HADOOP-7988. Upper case in hostname part of the principals doesn't work with kerberos.

          jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236960
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSecurityUtil.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1611 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1611/ ) HADOOP-7988 . Upper case in hostname part of the principals doesn't work with kerberos. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236960 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSecurityUtil.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Commit #432 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/432/)
          Merged r1236960 from trunk for HADOOP-7988.

          jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236962
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSecurityUtil.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #432 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/432/ ) Merged r1236960 from trunk for HADOOP-7988 . jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236962 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSecurityUtil.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-0.23-Commit #441 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/441/)
          Merged r1236960 from trunk for HADOOP-7988.

          jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236962
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSecurityUtil.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #441 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/441/ ) Merged r1236960 from trunk for HADOOP-7988 . jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236962 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSecurityUtil.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #1627 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1627/)
          HADOOP-7988. Upper case in hostname part of the principals doesn't work with kerberos.

          jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236960
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSecurityUtil.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1627 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1627/ ) HADOOP-7988 . Upper case in hostname part of the principals doesn't work with kerberos. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236960 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSecurityUtil.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Commit #456 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/456/)
          Merged r1236960 from trunk for HADOOP-7988.

          jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236962
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSecurityUtil.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #456 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/456/ ) Merged r1236960 from trunk for HADOOP-7988 . jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236962 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSecurityUtil.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #939 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/939/)
          HADOOP-7988. Upper case in hostname part of the principals doesn't work with kerberos.

          jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236960
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSecurityUtil.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #939 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/939/ ) HADOOP-7988 . Upper case in hostname part of the principals doesn't work with kerberos. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236960 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSecurityUtil.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Build #174 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/174/)
          Merged r1236960 from trunk for HADOOP-7988.

          jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236962
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSecurityUtil.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #174 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/174/ ) Merged r1236960 from trunk for HADOOP-7988 . jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236962 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSecurityUtil.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #152 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/152/)
          Merged r1236960 from trunk for HADOOP-7988.

          jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236962
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSecurityUtil.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #152 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/152/ ) Merged r1236960 from trunk for HADOOP-7988 . jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236962 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSecurityUtil.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #972 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/972/)
          HADOOP-7988. Upper case in hostname part of the principals doesn't work with kerberos.

          jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236960
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSecurityUtil.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #972 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/972/ ) HADOOP-7988 . Upper case in hostname part of the principals doesn't work with kerberos. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236960 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSecurityUtil.java
          Hide
          Matt Foley added a comment -

          Closed upon release 1.0.1.

          Show
          Matt Foley added a comment - Closed upon release 1.0.1.

            People

            • Assignee:
              Jitendra Nath Pandey
              Reporter:
              Jitendra Nath Pandey
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development