Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4324

JobClient can perhaps set mapreduce.job.credentials.binary rather than expect its presence?

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Won't Fix
    • 0.22.0, 2.0.0-alpha
    • None
    • mrv1, mrv2, security
    • None

    Description

      HDFS-1007 added in this requirement property "mapreduce.job.credentials.binary", that has lead Oozie to add the following duplicate snippet to all its Job-launching main classes such as the Pig, Hive, MR and Sqoop actions:

      if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") != null) {
                  jobConf.set("mapreduce.job.credentials.binary", System.getenv("HADOOP_TOKEN_FILE_LOCATION"));
      }
      

      Same is required for any client program that launches a job from within a task.

      Why can't this simply be set by the JobClient initialization bits itself? If no one imagines it causing issues, I'd like to add this snippet somewhere in JobSubmitter before it requests NN/JT, as otherwise we'd get…

      org.apache.hadoop.ipc.RemoteException: java.io.IOException: Delegation Token can be issued only with kerberos or web authentication 
      at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:5509) 
      at org.apache.hadoop.hdfs.server.namenode.NameNode.getDelegationToken(NameNode.java:536) 
      at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source) 
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
      at java.lang.reflect.Method.invoke(Method.java:597) 
      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557) 
      at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1434) 
      at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1430) 
      at java.security.AccessController.doPrivileged(Native Method) 
      at javax.security.auth.Subject.doAs(Subject.java:396) 
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157) 
      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1428)
      
      at org.apache.hadoop.ipc.Client.call(Client.java:1107) 
      at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:226) 
      at $Proxy6.getDelegationToken(Unknown Source) 
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) 
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
      at java.lang.reflect.Method.invoke(Method.java:597) 
      at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82) 
      at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59) 
      at $Proxy6.getDelegationToken(Unknown Source) 
      at org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:331) 
      at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(DistributedFileSystem.java:605) 
      at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:115) 
      at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:79) 
      at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:851) 
      at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:833) 
      at java.security.AccessController.doPrivileged(Native Method) 
      at javax.security.auth.Subject.doAs(Subject.java:396) 
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157) 
      at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:833) 
      at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:807) 
      at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1242) 
      

      … or similar errors when a user submits a job from a task running in a secured cluster.

      Let me know your thoughts on this!

      Attachments

        1. MAPREDUCE-4324.patch
          1.0 kB
          Harsh J

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            qwertymaniac Harsh J
            qwertymaniac Harsh J
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment