Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-3062

Fail to submit mapred job on a secured-HA-HDFS: logic URI cannot be picked up by job submission.

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 2.0.0-alpha
    • Component/s: ha, security
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      When testing the combination of NN HA + security + yarn, I found that the mapred job submission cannot pick up the logic URI of a nameservice.

      I have logic URI configured in core-site.xml

      <property>
       <name>fs.defaultFS</name>
       <value>hdfs://ns1</value>
      </property>
      

      HDFS client can work with the HA deployment/configs:

      [root@nn1 hadoop]# hdfs dfs -ls /
      Found 6 items
      drwxr-xr-x   - hbase  hadoop          0 2012-03-07 20:42 /hbase
      drwxrwxrwx   - yarn   hadoop          0 2012-03-07 20:42 /logs
      drwxr-xr-x   - mapred hadoop          0 2012-03-07 20:42 /mapred
      drwxr-xr-x   - mapred hadoop          0 2012-03-07 20:42 /mr-history
      drwxrwxrwt   - hdfs   hadoop          0 2012-03-07 21:57 /tmp
      drwxr-xr-x   - hdfs   hadoop          0 2012-03-07 20:42 /user
      

      but cannot submit a mapred job with security turned on

      [root@nn1 hadoop]# /usr/lib/hadoop/bin/yarn --config ./conf jar share/hadoop/mapreduce/hadoop-mapreduce-examples-0.24.0-SNAPSHOT.jar randomwriter out
      Running 0 maps.
      Job started: Wed Mar 07 23:28:23 UTC 2012
      java.lang.IllegalArgumentException: java.net.UnknownHostException: ns1
      	at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:431)
      	at org.apache.hadoop.security.SecurityUtil.buildDTServiceName(SecurityUtil.java:312)
      	at org.apache.hadoop.fs.FileSystem.getCanonicalServiceName(FileSystem.java:217)
      	at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:119)
      	at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:97)
      	at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:80)
      	at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:137)
      	at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:411)
      	at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:326)
      	at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1221)
      	at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218)
      ....
      

      0.24

        Attachments

        1. HDFS-3062-trunk.patch
          2 kB
          Mingjie Lai
        2. HDFS-3062-trunk-2.patch
          2 kB
          Mingjie Lai

          Issue Links

            Activity

              People

              • Assignee:
                mingjielai Mingjie Lai
                Reporter:
                mingjielai Mingjie Lai
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: