[MAPREDUCE-3669] Getting a lot of PriviledgedActionException / SaslException when running a job

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.23.0
• Fix Version/s: 0.23.1
• Component/s: mrv2
• Target Version/s: 0.23.1
• Assignee: Mahadev Konar
• Reporter: Thomas Graves
• Created: 13/Jan/12 20:16
• Updated: 05/Mar/12 02:48
• Resolved: 17/Jan/12 21:29

Description

On a secure cluster, when running a job we are seeing a lot of PriviledgedActionException / SaslExceptions. The job runs fine, its just the jobclient can't connect to the AM to get the progress information.

Its in a very tight loop retrying while getting the exceptions.

snip of the client log is:
12/01/13 15:33:45 INFO security.SecurityUtil: Acquired token Ident: 00 1c 68 61 64 6f 6f 70 71 61 40 44 45 56 2e 59 47
52 49 44 2e 59 41 48 4f 4f 2e 43 4f 4d 08 6d 61 70 72 65 64 71 61 00 8a 01 34 d7 b3 ff f5 8a 01 34 fb c0 83 f5 08 02,
Kind: HDFS_DELEGATION_TOKEN, Service: 10.10.10.10:8020
12/01/13 15:33:45 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 8 for user1 on 10.10.10.10:8020
12/01/13 15:33:45 INFO security.TokenCache: Got dt for
hdfs://host1.domain.com:8020;uri=10.10.10.10:8020;t.service=10.10.10.10:8020
12/01/13 15:33:45 WARN conf.Configuration: mapred.used.genericoptionsparser is deprecated. Instead, use
mapreduce.client.genericoptionsparser.used
12/01/13 15:33:45 INFO mapreduce.JobSubmitter: number of splits:2
12/01/13 15:33:45 INFO mapred.ResourceMgrDelegate: Submitted application application_1326410042859_0008 to
ResourceManager at rmhost.domain/10.10.10.11:8040
12/01/13 15:33:45 INFO mapreduce.Job: Running job: job_1326410042859_0008
12/01/13 15:33:52 INFO mapred.ClientServiceDelegate: The url to track the job:
rmhost.domain:8088/proxy/application_1326410042859_0008/
12/01/13 15:33:52 ERROR security.UserGroupInformation: PriviledgedActionException as:[e-mail]
(auth:SIMPLE) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail
ed to find any
Kerberos tgt)]
12/01/13 15:33:52 WARN ipc.Client: Exception encountered while connecting to the server :
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided
(Mechanism level: Failed to find any Kerberos tgt)]
12/01/13 15:33:52 ERROR security.UserGroupInformation: PriviledgedActionException as:[e-mail]
(auth:SIMPLE) cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (
Mechanism level:
Failed to find any Kerberos tgt)]
12/01/13 15:33:52 INFO mapred.ClientServiceDelegate: The url to track the job:
rmhost.domain:8088/proxy/application_1326410042859_0008/