Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3669

Getting a lot of PriviledgedActionException / SaslException when running a job

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.1
    • Component/s: mrv2
    • Labels:
      None

      Description

      On a secure cluster, when running a job we are seeing a lot of PriviledgedActionException / SaslExceptions. The job runs fine, its just the jobclient can't connect to the AM to get the progress information.

      Its in a very tight loop retrying while getting the exceptions.

      snip of the client log is:
      12/01/13 15:33:45 INFO security.SecurityUtil: Acquired token Ident: 00 1c 68 61 64 6f 6f 70 71 61 40 44 45 56 2e 59 47
      52 49 44 2e 59 41 48 4f 4f 2e 43 4f 4d 08 6d 61 70 72 65 64 71 61 00 8a 01 34 d7 b3 ff f5 8a 01 34 fb c0 83 f5 08 02,
      Kind: HDFS_DELEGATION_TOKEN, Service: 10.10.10.10:8020
      12/01/13 15:33:45 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 8 for user1 on 10.10.10.10:8020
      12/01/13 15:33:45 INFO security.TokenCache: Got dt for
      hdfs://host1.domain.com:8020;uri=10.10.10.10:8020;t.service=10.10.10.10:8020
      12/01/13 15:33:45 WARN conf.Configuration: mapred.used.genericoptionsparser is deprecated. Instead, use
      mapreduce.client.genericoptionsparser.used
      12/01/13 15:33:45 INFO mapreduce.JobSubmitter: number of splits:2
      12/01/13 15:33:45 INFO mapred.ResourceMgrDelegate: Submitted application application_1326410042859_0008 to
      ResourceManager at rmhost.domain/10.10.10.11:8040
      12/01/13 15:33:45 INFO mapreduce.Job: Running job: job_1326410042859_0008
      12/01/13 15:33:52 INFO mapred.ClientServiceDelegate: The url to track the job:
      rmhost.domain:8088/proxy/application_1326410042859_0008/
      12/01/13 15:33:52 ERROR security.UserGroupInformation: PriviledgedActionException as:user1@DEV.YGRID.YAHOO.COM
      (auth:SIMPLE) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail
      ed to find any
      Kerberos tgt)]
      12/01/13 15:33:52 WARN ipc.Client: Exception encountered while connecting to the server :
      javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided
      (Mechanism level: Failed to find any Kerberos tgt)]
      12/01/13 15:33:52 ERROR security.UserGroupInformation: PriviledgedActionException as:user1@DEV.YGRID.YAHOO.COM
      (auth:SIMPLE) cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (
      Mechanism level:
      Failed to find any Kerberos tgt)]
      12/01/13 15:33:52 INFO mapred.ClientServiceDelegate: The url to track the job:
      rmhost.domain:8088/proxy/application_1326410042859_0008/

        Attachments

        1. MAPREDUCE-3669.patch
          11 kB
          Mahadev konar
        2. MAPREDUCE-3669.patch
          39 kB
          Mahadev konar

          Activity

            People

            • Assignee:
              mahadev Mahadev konar
              Reporter:
              tgraves Thomas Graves
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: