Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-7733

Mapreduce jobs are failing when JT has hadoop.security.token.service.use_ip=false and client has hadoop.security.token.service.use_ip=true

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Won't Fix
    • 0.20.205.0
    • None
    • security
    • None

    Description

      I have added following property in core-site.xml of all the nodes in cluster and restarted

      <property>
      <name>hadoop.security.token.service.use_ip</name>
      <value>false</value>
      <description>desc</description>
      <final></final>
      </property>

      Then ran a randomwriter, distcp jobs, they are all failing
      $HADOOP_HOME/bin/hadoop --config $HADOOP_CONFIG_DIR jar $HADOOP_HOME/hadoop-examples.jar randomwriter -Dtest.randomwrite.bytes_per_map=256000 input_1318325953
      Running 140 maps.
      Job started: Tue Oct 11 09:48:09 UTC 2011
      11/10/11 09:48:09 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 14 for <USERNAME> on <Namenode IP>:8020
      11/10/11 09:48:09 INFO security.TokenCache: Got dt for
      hdfs://<Namenode Hostname>/user/<USERNAME>/.staging/job_201110110946_0001;uri=<Namenode IP>:8020;t.service=<Namenode IP>:8020
      11/10/11 09:48:09 INFO mapred.JobClient: Cleaning up the staging area
      hdfs://<Namenode Hostname>/user/<USERNAME>/.staging/job_201110110946_0001
      org.apache.hadoop.ipc.RemoteException: java.io.IOException: java.io.IOException: Call to
      <Namenode Hostname>/<Namenode IP>:8020 failed on local exception: java.io.IOException:
      javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided
      (Mechanism level: Failed to find any Kerberos tgt)]
      at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3943)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)
      at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)
      at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:396)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382)
      Caused by: java.io.IOException: Call to <Namenode Hostname>/<Namenode IP>:8020 failed on local exception:
      java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid
      credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
      at org.apache.hadoop.ipc.Client.wrapException(Client.java:1103)
      at org.apache.hadoop.ipc.Client.call(Client.java:1071)
      at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
      at $Proxy7.getProtocolVersion(Unknown Source)
      at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
      at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
      at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:118)
      at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:222)
      at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:187)
      at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
      at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1328)
      at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:65)
      at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1346)
      at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:244)
      at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187)
      at org.apache.hadoop.mapred.JobInProgress$2.run(JobInProgress.java:401)
      at org.apache.hadoop.mapred.JobInProgress$2.run(JobInProgress.java:399)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:396)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
      at org.apache.hadoop.mapred.JobInProgress.<init>(JobInProgress.java:399)
      at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3941)
      ... 11 more
      Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No
      valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
      at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:539)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:396)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
      at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:484)
      at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:586)
      at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:184)
      at org.apache.hadoop.ipc.Client.getConnection(Client.java:1202)
      at org.apache.hadoop.ipc.Client.call(Client.java:1046)
      ... 31 more
      Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials
      provided (Mechanism level: Failed to find any Kerberos tgt)]
      at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:194)
      at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:134)
      at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:381)
      at org.apache.hadoop.ipc.Client$Connection.access$1100(Client.java:184)
      at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:579)
      at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:576)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:396)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
      at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:575)
      ... 34 more
      Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
      at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:130)
      at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:106)
      at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:172)
      at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:209)
      at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:195)
      at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:162)
      at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:175)
      ... 43 more

      at org.apache.hadoop.ipc.Client.call(Client.java:1066)
      at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
      at org.apache.hadoop.mapred.$Proxy7.submitJob(Unknown Source)
      at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:913)
      at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:842)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:396)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
      at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:842)
      at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:816)
      at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1253)
      at org.apache.hadoop.examples.RandomWriter.run(RandomWriter.java:272)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
      at org.apache.hadoop.examples.RandomWriter.main(RandomWriter.java:283)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
      at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
      at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

      At the sametime Jobtracker log says
      ===================================
      2011-10-11 09:48:09,486 WARN org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server :
      javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided
      (Mechanism level: Failed to find any Kerberos tgt)]
      2011-10-11 09:48:09,487 INFO org.apache.hadoop.ipc.Server: IPC Server handler 26 on 50300, call
      submitJob(job_201110110946_0001, hdfs://<Namenode hostname> /user/<USER>/.staging/job_201110110946_0001,
      org.apache.hadoop.security.Credentials@16381a53) from <client IP> :46859: error: java.io.IOException:
      java.io.IOException: Call to <Namenode hostname> /<Namenode IP> :8020 failed on local exception:
      java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid
      credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
      java.io.IOException: java.io.IOException: Call to <Namenode hostname> /<Namenode IP> :8020 failed on local
      exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No
      valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
      at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3943)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)
      at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)
      at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:396)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382)
      Caused by: java.io.IOException: Call to <Namenode hostname> /<Namenode IP> :8020 failed on local exception:
      java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid
      credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
      at org.apache.hadoop.ipc.Client.wrapException(Client.java:1103)
      at org.apache.hadoop.ipc.Client.call(Client.java:1071)
      at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
      at $Proxy7.getProtocolVersion(Unknown Source)
      at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
      at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
      at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:118)
      at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:222)
      at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:187)
      at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
      at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1328)
      at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:65)
      at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1346)
      at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:244)
      at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187)
      at org.apache.hadoop.mapred.JobInProgress$2.run(JobInProgress.java:401)
      at org.apache.hadoop.mapred.JobInProgress$2.run(JobInProgress.java:399)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:396)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
      at org.apache.hadoop.mapred.JobInProgress.<init>(JobInProgress.java:399)
      at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3941)
      ... 11 more

      Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No
      valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
      at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:539)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:396)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
      at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:484)
      at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:586)
      at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:184)
      at org.apache.hadoop.ipc.Client.getConnection(Client.java:1202)
      at org.apache.hadoop.ipc.Client.call(Client.java:1046)
      ... 31 more
      Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials
      provided (Mechanism level: Failed to find any Kerberos tgt)]
      at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:194)
      at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:134)
      at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:381)
      at org.apache.hadoop.ipc.Client$Connection.access$1100(Client.java:184)
      at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:579)
      at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:576)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:396)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
      at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:575)
      ... 34 more
      Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
      at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:130)
      at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:106)
      at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:172)
      at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:209)
      at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:195)
      at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:162)
      at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:175)
      ... 43 more

      Attachments

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-13206 Delegation token cannot be fetched and used by different versions of client

          • Major - Major loss of function.
          • Patch Available

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-12665 Document hadoop.security.token.service.use_ip

          • Major - Major loss of function.
          • Resolved
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-6565 Configuration to use host name in delegation token service is not read from job.xml during MapReduce job execution.

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              daryn Daryn Sharp
              rajsaha Rajit Saha
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: