Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
0.20.205.0
-
None
-
None
Description
Using the FairScheduler with kerberos authentication does not appear to work with 0.20.205.0. When submitting a job, execution fails when storing the delegation tokens for the job:
11/10/18 21:48:53 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 6 for hdfs on xxxx:8020 11/10/18 21:48:53 INFO security.TokenCache: Got dt for hdfs://xxxx/user/hdfs/.staging/job_201110182148_0001;uri=xxxx:8020;t.service=xxxx:8020 11/10/18 21:48:53 INFO mapred.FileInputFormat: Total input paths to process : 1 11/10/18 21:48:53 INFO mapred.JobClient: Running job: job_201110182148_0001 11/10/18 21:48:54 INFO mapred.JobClient: map 0% reduce 0% 11/10/18 21:48:54 INFO mapred.JobClient: Job complete: job_201110182148_0001 11/10/18 21:48:54 INFO mapred.JobClient: Counters: 0 11/10/18 21:48:54 INFO mapred.JobClient: Job Failed: Job initialization failed: java.io.IOException: Call to xxxx/xxxx:8020 failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Attempt to obtain new INITIATE credentials failed! (null))] at org.apache.hadoop.ipc.Client.wrapException(Client.java:1103) at org.apache.hadoop.ipc.Client.call(Client.java:1071) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225) at $Proxy7.getProtocolVersion(Unknown Source) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379) at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:118) at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:222) at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:187) at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1328) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:65) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1346) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:244) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187) at org.apache.hadoop.security.Credentials.writeTokenStorageFile(Credentials.java:175) at org.apache.hadoop.mapred.JobInProgress.generateAndStoreTokens(JobInProgress.java:3528) at org.apache.hadoop.mapred.JobInProgress.initTasks(JobInProgress.java:696) at org.apache.hadoop.mapred.JobTracker.initJob(JobTracker.java:4207) at org.apache.hadoop.mapred.FairScheduler$JobInitializer$InitJob.run(FairScheduler.java:291) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Attempt to obtain new INITIATE credentials failed! (null))] at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:539) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059) at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:484) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:586) at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:184) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1202) at org.apache.hadoop.ipc.Client.call(Client.java:1046) ... 21 more Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Attempt to obtain new INITIATE credentials failed! (null))] at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:194) at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:134) at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:381) at org.apache.hadoop.ipc.Client$Connection.access$1100(Client.java:184) at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:579) at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:576) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:575) ... 24 more Caused by: GSSException: No valid credentials provided (Mechanism level: Attempt to obtain new INITIATE credentials failed! (null)) at sun.security.jgss.krb5.Krb5InitCredential.getTgt(Krb5InitCredential.java:333) at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:128) at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:106) at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:172) at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:209) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:195) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:162) at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:175) ... 33 more Caused by: javax.security.auth.login.LoginException: No LoginModules configured for at javax.security.auth.login.LoginContext.init(LoginContext.java:256) at javax.security.auth.login.LoginContext.<init>(LoginContext.java:499) at sun.security.jgss.GSSUtil.login(GSSUtil.java:244) at sun.security.jgss.krb5.Krb5Util.getTicket(Krb5Util.java:136) at sun.security.jgss.krb5.Krb5InitCredential$1.run(Krb5InitCredential.java:328) at java.security.AccessController.doPrivileged(Native Method) at sun.security.jgss.krb5.Krb5InitCredential.getTgt(Krb5InitCredential.java:325) ... 40 more
The problem seems to have been introduced by the backported changes in MAPREDUCE-2981, which shifted the execution of JobTracker.initJob(), and hence JobInProgress.generateAndStoreTokens(), to underneath the call path for the RPC invocation. As a result, the DFS write in TokenStorage.writeTokenStorageFile() in done under a UGI.doAs() block as the RPC client remote user, without a TGT for negotiating the connection.
Does this analysis seem right? Previously it seems that JobTracker.initJob() was only called in a separate thread so it was picking up the credentials obtained for the configured JobTracker kerberos principal. The same job runs successfully in a build with MAPREDUCE-2981 reverted.
Attachments
Attachments
Issue Links
- relates to
-
MAPREDUCE-2981 Backport trunk fairscheduler to 0.20-security branch
- Closed