Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5375

Delegation Token renewal exception in jobtracker logs

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 1.2.0
    • Fix Version/s: 1.2.1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Filing on behalf of Venkat Ranganathan who found this originally and provided a patch.

      Saw this in the JT logs while oozie tests were running with Hadoop.

      When Oozie java action is executed, the following shows up in the job tracker log.

      ERROR org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal: Exception renewing tokenIdent: 00 07 68 64 70 75 73 65 72 06 6d 61 70 72 65 64 26 6f 6f 7a 69 65 2f 63 6f 6e 64 6f 72 2d 73 65 63 2e 76 65 6e 6b 61 74 2e 6f 72 67 40 76 65 6e 6b 61 74 2e 6f 72 67 8a 01 3e a6 87 5e 5b 8a 01 3e ca 93 e2 5b 02 02, Kind: MAPREDUCE_DELEGATION_TOKEN, Service: ip:50300. Not rescheduled
      org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.security.AccessControlException: Client jt/host@domain.com tries to renew a token with renewer specified as mapred
              at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.renewToken(AbstractDelegationTokenSecretManager.java:267)
              at org.apache.hadoop.mapred.JobTracker.renewDelegationToken(JobTracker.java:3878)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
              at java.lang.reflect.Method.invoke(Method.java:597)
              at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
              at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1405)
              at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1401)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:396)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
              at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1399)
      
              at org.apache.hadoop.ipc.Client.call(Client.java:1118)
              at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
              at org.apache.hadoop.mapred.$Proxy8.renewDelegationToken(Unknown Source)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
              at java.lang.reflect.Method.invoke(Method.java:597)
              at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
             at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
              at org.apache.hadoop.mapred.$Proxy8.renewDelegationToken(Unknown Source)
              at org.apache.hadoop.mapred.JobClient$Renewer.renew(JobClient.java:578)
              at org.apache.hadoop.security.token.Token.renew(Token.java:309)
              at org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$RenewalTimerTask$1.run(DelegationTokenRenewal.java:221)
              at org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$RenewalTimerTask$1.run(DelegationTokenRenewal.java:217)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:396)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
              at org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$RenewalTimerTask.run(DelegationTokenRenewal.java:216)
              at java.util.TimerThread.mainLoop(Timer.java:512)
              at java.util.TimerThread.run(Timer.java:462)
      

      Setting the renewer to Kerberos Local name does not help because AbstractDelegationTokenIdentifier sets the renewer to Kerberos shortname but JobTracker.renewDelegationToken uses the fullName. This essentially causes the renewal to fail.

      1. MRDelegationIssue-MR-5375.patch
        0.7 kB
        Vinod Kumar Vavilapalli

        Issue Links

          Activity

          Hide
          Vinod Kumar Vavilapalli added a comment -

          Debugged this with Venkat Ranganathan and Siddharth Seth. There are a few issues here.

          One is the HDFS delegation token renewal issue. Oozie does not handle the auth to locals mapping configuration correctly. Furthermore, there can be multiple version of this auth to local configuration in different clusters that Oozie has (specified by the oozie configuration property oozie.service.HadoopAccessorService.hadoop.configurations). The entries are keyed by the host:port of the jt/nn host:port but there is a default catch-all configuration if specific hadoop configuration is not specified. The fix on oozie side is to use the oozie catch-all configuration (specified by *= in the oozie hadoop config mapping entry) to have the right auth to local mapping and use it in all the clusters that oozie is launching jobs.

          There is MR side delegation issue to. Irrespective of the renewer principal Oozie sets, the JobTracker changes the token to a local name but then expects it to be a full principal when renewwing. So, obviously Oozie cannot satisfy this requirement and the MR delegation issue has to be fixed in Jobtracker and the HDFS delegation issue will be addressed in Oozie.

          Show
          Vinod Kumar Vavilapalli added a comment - Debugged this with Venkat Ranganathan and Siddharth Seth . There are a few issues here. One is the HDFS delegation token renewal issue. Oozie does not handle the auth to locals mapping configuration correctly. Furthermore, there can be multiple version of this auth to local configuration in different clusters that Oozie has (specified by the oozie configuration property oozie.service.HadoopAccessorService.hadoop.configurations). The entries are keyed by the host:port of the jt/nn host:port but there is a default catch-all configuration if specific hadoop configuration is not specified. The fix on oozie side is to use the oozie catch-all configuration (specified by *= in the oozie hadoop config mapping entry) to have the right auth to local mapping and use it in all the clusters that oozie is launching jobs. There is MR side delegation issue to. Irrespective of the renewer principal Oozie sets, the JobTracker changes the token to a local name but then expects it to be a full principal when renewwing. So, obviously Oozie cannot satisfy this requirement and the MR delegation issue has to be fixed in Jobtracker and the HDFS delegation issue will be addressed in Oozie.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Here's a patch from Venkat Ranganathan. With this patch and with the fix for Oozie auth_to_local issue, there are no more exceptions in the jobtracker log complaining about delegation renewal issues.

          Show
          Vinod Kumar Vavilapalli added a comment - Here's a patch from Venkat Ranganathan . With this patch and with the fix for Oozie auth_to_local issue, there are no more exceptions in the jobtracker log complaining about delegation renewal issues.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Obviously I am +1 for the patch. Tx to Siddharth Seth for the discussion to corner down the real fix.

          Show
          Vinod Kumar Vavilapalli added a comment - Obviously I am +1 for the patch. Tx to Siddharth Seth for the discussion to corner down the real fix.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12590921/MRDelegationIssue-MR-5375.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3832//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12590921/MRDelegationIssue-MR-5375.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3832//console This message is automatically generated.
          Hide
          Venkat Ranganathan added a comment -

          Thanks Vinod Kumar Vavilapalli and Siddharth Seth for help with understanding and fixing this issue. I had created MAPREDUCE-5249 - Resolved that was duplicate.

          Show
          Venkat Ranganathan added a comment - Thanks Vinod Kumar Vavilapalli and Siddharth Seth for help with understanding and fixing this issue. I had created MAPREDUCE-5249 - Resolved that was duplicate.
          Hide
          Arun C Murthy added a comment -

          I just committed this. Thanks Venkat, Vinod & Sid!

          Show
          Arun C Murthy added a comment - I just committed this. Thanks Venkat, Vinod & Sid!

            People

            • Assignee:
              Venkat Ranganathan
              Reporter:
              Venkat Ranganathan
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development