Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3669

Getting a lot of PriviledgedActionException / SaslException when running a job

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.1
    • Component/s: mrv2
    • Labels:
      None

      Description

      On a secure cluster, when running a job we are seeing a lot of PriviledgedActionException / SaslExceptions. The job runs fine, its just the jobclient can't connect to the AM to get the progress information.

      Its in a very tight loop retrying while getting the exceptions.

      snip of the client log is:
      12/01/13 15:33:45 INFO security.SecurityUtil: Acquired token Ident: 00 1c 68 61 64 6f 6f 70 71 61 40 44 45 56 2e 59 47
      52 49 44 2e 59 41 48 4f 4f 2e 43 4f 4d 08 6d 61 70 72 65 64 71 61 00 8a 01 34 d7 b3 ff f5 8a 01 34 fb c0 83 f5 08 02,
      Kind: HDFS_DELEGATION_TOKEN, Service: 10.10.10.10:8020
      12/01/13 15:33:45 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 8 for user1 on 10.10.10.10:8020
      12/01/13 15:33:45 INFO security.TokenCache: Got dt for
      hdfs://host1.domain.com:8020;uri=10.10.10.10:8020;t.service=10.10.10.10:8020
      12/01/13 15:33:45 WARN conf.Configuration: mapred.used.genericoptionsparser is deprecated. Instead, use
      mapreduce.client.genericoptionsparser.used
      12/01/13 15:33:45 INFO mapreduce.JobSubmitter: number of splits:2
      12/01/13 15:33:45 INFO mapred.ResourceMgrDelegate: Submitted application application_1326410042859_0008 to
      ResourceManager at rmhost.domain/10.10.10.11:8040
      12/01/13 15:33:45 INFO mapreduce.Job: Running job: job_1326410042859_0008
      12/01/13 15:33:52 INFO mapred.ClientServiceDelegate: The url to track the job:
      rmhost.domain:8088/proxy/application_1326410042859_0008/
      12/01/13 15:33:52 ERROR security.UserGroupInformation: PriviledgedActionException as:user1@DEV.YGRID.YAHOO.COM
      (auth:SIMPLE) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail
      ed to find any
      Kerberos tgt)]
      12/01/13 15:33:52 WARN ipc.Client: Exception encountered while connecting to the server :
      javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided
      (Mechanism level: Failed to find any Kerberos tgt)]
      12/01/13 15:33:52 ERROR security.UserGroupInformation: PriviledgedActionException as:user1@DEV.YGRID.YAHOO.COM
      (auth:SIMPLE) cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (
      Mechanism level:
      Failed to find any Kerberos tgt)]
      12/01/13 15:33:52 INFO mapred.ClientServiceDelegate: The url to track the job:
      rmhost.domain:8088/proxy/application_1326410042859_0008/

      1. MAPREDUCE-3669.patch
        39 kB
        Mahadev konar
      2. MAPREDUCE-3669.patch
        11 kB
        Mahadev konar

        Activity

        Thomas Graves created issue -
        Vinod Kumar Vavilapalli made changes -
        Field Original Value New Value
        Assignee Mahadev konar [ mahadev ]
        Mahadev konar made changes -
        Attachment MAPREDUCE-3669.patch [ 12510638 ]
        Mahadev konar made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Mahadev konar made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Mahadev konar made changes -
        Attachment MAPREDUCE-3669.patch [ 12510793 ]
        Mahadev konar made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Arun C Murthy made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Vinod Kumar Vavilapalli made changes -
        Fix Version/s 0.23.1 [ 12318883 ]
        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Mahadev konar
            Reporter:
            Thomas Graves
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development