Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5375

Delegation Token renewal exception in jobtracker logs

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 1.2.0
    • Fix Version/s: 1.2.1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Filing on behalf of Venkat Ranganathan who found this originally and provided a patch.

      Saw this in the JT logs while oozie tests were running with Hadoop.

      When Oozie java action is executed, the following shows up in the job tracker log.

      ERROR org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal: Exception renewing tokenIdent: 00 07 68 64 70 75 73 65 72 06 6d 61 70 72 65 64 26 6f 6f 7a 69 65 2f 63 6f 6e 64 6f 72 2d 73 65 63 2e 76 65 6e 6b 61 74 2e 6f 72 67 40 76 65 6e 6b 61 74 2e 6f 72 67 8a 01 3e a6 87 5e 5b 8a 01 3e ca 93 e2 5b 02 02, Kind: MAPREDUCE_DELEGATION_TOKEN, Service: ip:50300. Not rescheduled
      org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.security.AccessControlException: Client jt/host@domain.com tries to renew a token with renewer specified as mapred
              at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.renewToken(AbstractDelegationTokenSecretManager.java:267)
              at org.apache.hadoop.mapred.JobTracker.renewDelegationToken(JobTracker.java:3878)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
              at java.lang.reflect.Method.invoke(Method.java:597)
              at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
              at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1405)
              at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1401)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:396)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
              at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1399)
      
              at org.apache.hadoop.ipc.Client.call(Client.java:1118)
              at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
              at org.apache.hadoop.mapred.$Proxy8.renewDelegationToken(Unknown Source)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
              at java.lang.reflect.Method.invoke(Method.java:597)
              at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
             at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
              at org.apache.hadoop.mapred.$Proxy8.renewDelegationToken(Unknown Source)
              at org.apache.hadoop.mapred.JobClient$Renewer.renew(JobClient.java:578)
              at org.apache.hadoop.security.token.Token.renew(Token.java:309)
              at org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$RenewalTimerTask$1.run(DelegationTokenRenewal.java:221)
              at org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$RenewalTimerTask$1.run(DelegationTokenRenewal.java:217)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:396)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
              at org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$RenewalTimerTask.run(DelegationTokenRenewal.java:216)
              at java.util.TimerThread.mainLoop(Timer.java:512)
              at java.util.TimerThread.run(Timer.java:462)
      

      Setting the renewer to Kerberos Local name does not help because AbstractDelegationTokenIdentifier sets the renewer to Kerberos shortname but JobTracker.renewDelegationToken uses the fullName. This essentially causes the renewal to fail.

      1. MRDelegationIssue-MR-5375.patch
        0.7 kB
        Vinod Kumar Vavilapalli

        Issue Links

          Activity

          No work has yet been logged on this issue.

            People

            • Assignee:
              Venkat Ranganathan
              Reporter:
              Venkat Ranganathan
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development