Details
Description
Kerberos doesn't like upper case in the hostname part of the principals.
This issue has been seen in 23 as well as 1.0.
Attachments
Attachments
- HADOOP-7988.branch-1.patch
- 2 kB
- Jitendra Nath Pandey
- HADOOP-7988.branch-1.patch
- 0.6 kB
- Jitendra Nath Pandey
- HADOOP-7988.trunk.patch
- 2 kB
- Jitendra Nath Pandey
Issue Links
- is related to
-
KUDU-1942 Kerberos fails to log in on hostnames with capital characters
- Resolved
-
HADOOP-8878 uppercase namenode hostname causes hadoop dfs calls with webhdfs filesystem and fsck to fail when security is on
- Closed
-
HADOOP-8882 uppercase namenode host name causes fsck to fail when useKsslAuth is on
- Closed
- relates to
-
AMBARI-2820 In secured cluster enabling wizard, hostname part in the principal should be displayed in lower case format
- Resolved
Activity
It is expected that principals will be created with lower case in the hostname in the kdc. The patch makes sure, that lower case hostname is used in the principal. The patch has been tested on machines having upper case in their hostnames.
I'm -1 this patch without a config option to turn this behavior off.
Oh, and before I forget, this should not be the default behavior.
The issue this jira is trying to fix exists in both 0.23 and branch-1.
Following excpetion was observed in 0.23, when we have upper case in the principals.
2012-01-20 12:03:08,258 INFO ipc.HadoopYarnRPC (HadoopYarnProtoRPC.java:getProxy(48)) - Creating a HadoopYarnProtoRpc proxy for protocol interface org.apache.hadoop.yarn.server.api.ResourceTracker
2012-01-20 12:03:08,291 INFO nodemanager.NodeStatusUpdaterImpl (NodeStatusUpdaterImpl.java:registerWithRM(155)) - Connected to ResourceManager at hadoopRM.example.aurora:9003
2012-01-20 12:03:20,399 WARN ipc.Client (Client.java:run(526)) - Couldn't setup connection fornm/hadoopNM.example.aurora@EXAMPLE.AURORA to rm/hadoopRM.example.aurora@EXAMPLE.AURORA
2012-01-20 12:03:20,405 ERROR service.CompositeService (CompositeService.java:start(72)) - Error starting services org.apache.hadoop.yarn.server.nodemanager.NodeManager
org.apache.avro.AvroRuntimeException: java.lang.reflect.UndeclaredThrowableException
at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.start(NodeStatusUpdaterImpl.java:132)
at org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.start(NodeManager.java:163)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:231)
Caused by: java.lang.reflect.UndeclaredThrowableException
at org.apache.hadoop.yarn.server.api.impl.pb.client.ResourceTrackerPBClientImpl.registerNodeManager(ResourceTrackerPBClientImpl.java:66)
at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.registerWithRM(NodeStatusUpdaterImpl.java:161)
at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.start(NodeStatusUpdaterImpl.java:128)
... 3 more
Caused by: com.google.protobuf.ServiceException: java.io.IOException: Failed on local exception: java.io.IOException: Couldn't setup connection for nm/hadoopNM.example.aurora@EXAMPLE.AURORA to rm/hadoopRM.example.aurora@EXAMPLE.AURORA; Host Details : local host is: "hadoopNM/10.112.127.102"; destination host is: ""hadoopRM.example.aurora":9003;
at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:139)
at $Proxy14.registerNodeManager(Unknown Source)
at org.apache.hadoop.yarn.server.api.impl.pb.client.ResourceTrackerPBClientImpl.registerNodeManager(ResourceTrackerPBClientImpl.java:59)
... 5 more
Caused by: java.io.IOException: Failed on local exception: java.io.IOException: Couldn't setup connection fornm/hadoopNM.example.aurora@EXAMPLE.AURORA to rm/hadoopRM.example.aurora@EXAMPLE.AURORA; Host Details : local host is: "hadoopNM/10.112.127.102"; destination host is: ""hadoopRM.example.aurora":9003;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:655)
at org.apache.hadoop.ipc.Client.call(Client.java:1089)
at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:136)
... 7 more
Caused by: java.io.IOException: Couldn't setup connection for nm/hadoopNM.example.aurora@EXAMPLE.AURORA torm/hadoopRM.example.aurora@EXAMPLE.AURORA
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:527)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1152)
at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:499)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:583)
at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:205)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1195)
at org.apache.hadoop.ipc.Client.call(Client.java:1065)
... 8 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - UNKNOWN_SERVER)]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:194)
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:137)
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:407)
at org.apache.hadoop.ipc.Client$Connection.access$1200(Client.java:205)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:576)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:573)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1152)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:572)
... 11 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - UNKNOWN_SERVER)
at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:663)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:230)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:162)
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:175)
... 20 more
Caused by: KrbException: Server not found in Kerberos database (7) - UNKNOWN_SERVER
at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:64)
at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:185)
at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:294)
at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:106)
at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:557)
at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:594)
... 23 more
Caused by: KrbException: Identifier doesn't match expected value (906)
at sun.security.krb5.internal.KDCRep.init(KDCRep.java:133)
at sun.security.krb5.internal.TGSRep.init(TGSRep.java:58)
at sun.security.krb5.internal.TGSRep.<init>(TGSRep.java:53)
at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:46)
... 28 more
The patch only tries to standardize that hadoop always uses lowercase in the hostname part of the kerberos principal, making sure that kerberos always sees lower case in the hostname part of the principals. The patch doesn't change the hostname anywhere, only the principals used are affected.
Just for reference of a similar code, hbase code uses same SecurityUtil api and explicitly converts the hostname to lowercase.
Ask me if I care what HBase does.
However, I did have a chat with a friend of mine who reminded me that RFC 4150 (which I completely forgot about) actually does clarify the previous, murky rules around server principals by saying that the host component for case insensitive systems should be forced lower case. Note that this is for server principals. We are back in slightly murky waters again when you consider KSSL and SPNEGO....
But even so, I'll remove my -1.
This is a complex topic. I'll try to summarize what I've found, but I'll also give the full reference hyperlinks.
The Kerberos RFC says this:
6.2.1. Name of Server Principals
... The first component of the
two- or multi-component name will identify the service, and the
latter components will identify the host. Where the name of the host
is not case sensitive (for example, with Internet domain names) the
name of the host MUST be lowercase.
The Oracle Java Docs say this: Kerberos Requirements
By convention, all Kerberos realm names are uppercase and all DNS hostname and domain names are lowercase. On the Windows 2000 platform, domains are also Kerberos realms; however, the realm name is always the uppercase version of the domain name.
Hostnames are case insensitive and by convention they are all lowercase. They must resolve to the same hostname on the client and server by their respective naming services.
However, in the Kerberos database hostnames are case sensitive. In all host-based Kerberos service principals in the KDC, hostnames are case-sensitive. The hostnames used in the Kerberos service principal names must exactly match the hostnames returned by the naming service. For example, if the naming service returns a fully qualified lowercased DNS hostname, such as "raven.sun.com", then the administrator must use the same fully qualified lowercased DNS hostname when creating host-based principal names in the KDC: "host/raven.sun.com".
Finally, in an older IETF Draft related to Distributing Kerberos KDC and Realm Information with DNS the following commentary is provided:
DNS vs. Kerberos - Case Sensitivity of Realm Names
In Kerberos, realm names are case sensitive. While it is strongly
encouraged that all realm names be all upper case this recommendation
has not been adopted by all sites. Some sites use all lower case
names and other use mixed case. DNS on the other hand is case insen-
sitive for queries but is case preserving for responses to TXT
queries. Since "MYREALM", "myrealm", and "MyRealm" are all different
it is necessary that only one of the possible combinations of upper
and lower case characters be used. This restriction may be lifted in
the future as the DNS naming scheme is expanded to support non-ASCII
names.
These are the only authoritative references I could find. However, there is a considerable body of archived email list correspondence that strongly implies the Java Kerberos packages work best if host names are forced to lower case in the Kerberos principle names.
The key issue seems to be the statement in the second reference, "The hostnames used in the Kerberos service principal names must exactly match the hostnames returned by the naming service," plus the observation in the third reference, {{ "it is necessary that only one of the possible combinations of upper and lower case characters be used [the same way in Kerberos and DNS]." }}
So, if DNS servers and Kerberos servers disagree on hostname case usage and/or sensitivity, authentication won't work. For example, see this detailed complaint from 2010, mixed case hostname issue , where the response from a Kerberos contributor was simply "Hostnames are always case folded (to lower case) in principal names."
There are also threads that imply the Java Kerberos packages may have or had bugs around this issue that are resolved by forcing hostnames to lower case. For example, see
http://www.nexentastor.org/projects/site/wiki/CIFS :
There is a known bug in the kerberos package which breaks ticketing in Nexenta if your domain name (in the case of active directory users) is mixed case or all upper case. Because kerberos requires all lower case this causes ticket errors that can be hard to trace down.
and http://linux-nfs.org/pipermail/nfsv4/2005-July/002278.html :
Browsing the code showed that handling keytab file has been done in
different ways in Server and Client. While the server code mandates
hostnames/principals to be of lowercase, client code doesn't impose any
restriction...
The current theory on the client was that we would use any usable
key we found to do the mount. It can be argued that with
authenticated delegation callback, the client becomes a server
and therefore the server principal name restriction should apply.
If everyone agrees with that, I will make the changes to restrict
the principal name to be used on the client.
To find other references, search the Web for "kerberos mixed case" and "kerberos requires lowercase".
It seems to me that given the above, it makes sense that both HBase and Hadoop would choose to fold hostnames to lower case, in the Kerberos principals only.
So, if DNS servers and Kerberos servers disagree on hostname case usage and/or sensitivity, authentication won't work.
Right. Which is why prior to 4150, forcing case at the app layer could result in all sorts of surprising behavior since, as you pointed out, it was ill-defined. The expectation was that the Kerberos principals would be set up to match DNS in those cases where DNS itself didn't return lower case. So if your DNS server responded with hostname.Example.ORG, then your host principal would be host/hostname.Example.ORG. From an administrative perspective, this was worked around by including both the "real" name and the folded name principals in the realm and associated keytabs.
Thanks, @Allen.
@Jitendra, please provide a unit test, then provide patch for trunk and get auto-test to run.
Thanks.
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12512250/HADOOP-7988.trunk.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/544//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/544//console
This message is automatically generated.
+1 for code review, both trunk and branch-1 versions. Thanks, Jitendra!
Integrated in Hadoop-Hdfs-trunk-Commit #1683 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1683/)
HADOOP-7988. Upper case in hostname part of the principals doesn't work with kerberos.
jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236960
Files :
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSecurityUtil.java
Integrated in Hadoop-Common-trunk-Commit #1611 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1611/)
HADOOP-7988. Upper case in hostname part of the principals doesn't work with kerberos.
jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236960
Files :
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSecurityUtil.java
Integrated in Hadoop-Hdfs-0.23-Commit #432 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/432/)
Merged r1236960 from trunk for HADOOP-7988.
jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236962
Files :
- /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
- /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
- /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSecurityUtil.java
Integrated in Hadoop-Common-0.23-Commit #441 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/441/)
Merged r1236960 from trunk for HADOOP-7988.
jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236962
Files :
- /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
- /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
- /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSecurityUtil.java
Integrated in Hadoop-Mapreduce-trunk-Commit #1627 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1627/)
HADOOP-7988. Upper case in hostname part of the principals doesn't work with kerberos.
jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236960
Files :
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSecurityUtil.java
Integrated in Hadoop-Mapreduce-0.23-Commit #456 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/456/)
Merged r1236960 from trunk for HADOOP-7988.
jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236962
Files :
- /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
- /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
- /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSecurityUtil.java
Integrated in Hadoop-Hdfs-trunk #939 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/939/)
HADOOP-7988. Upper case in hostname part of the principals doesn't work with kerberos.
jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236960
Files :
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSecurityUtil.java
Integrated in Hadoop-Mapreduce-0.23-Build #174 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/174/)
Merged r1236960 from trunk for HADOOP-7988.
jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236962
Files :
- /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
- /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
- /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSecurityUtil.java
Integrated in Hadoop-Hdfs-0.23-Build #152 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/152/)
Merged r1236960 from trunk for HADOOP-7988.
jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236962
Files :
- /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
- /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
- /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSecurityUtil.java
Integrated in Hadoop-Mapreduce-trunk #972 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/972/)
HADOOP-7988. Upper case in hostname part of the principals doesn't work with kerberos.
jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236960
Files :
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSecurityUtil.java
Kerberos is case sensitive. So if the local hostname returns upper case, this patch will likely break that machine from working.