Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-3062

Fail to submit mapred job on a secured-HA-HDFS: logic URI cannot be picked up by job submission.

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 2.0.0-alpha
    • Component/s: ha, security
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      When testing the combination of NN HA + security + yarn, I found that the mapred job submission cannot pick up the logic URI of a nameservice.

      I have logic URI configured in core-site.xml

      <property>
       <name>fs.defaultFS</name>
       <value>hdfs://ns1</value>
      </property>
      

      HDFS client can work with the HA deployment/configs:

      [root@nn1 hadoop]# hdfs dfs -ls /
      Found 6 items
      drwxr-xr-x   - hbase  hadoop          0 2012-03-07 20:42 /hbase
      drwxrwxrwx   - yarn   hadoop          0 2012-03-07 20:42 /logs
      drwxr-xr-x   - mapred hadoop          0 2012-03-07 20:42 /mapred
      drwxr-xr-x   - mapred hadoop          0 2012-03-07 20:42 /mr-history
      drwxrwxrwt   - hdfs   hadoop          0 2012-03-07 21:57 /tmp
      drwxr-xr-x   - hdfs   hadoop          0 2012-03-07 20:42 /user
      

      but cannot submit a mapred job with security turned on

      [root@nn1 hadoop]# /usr/lib/hadoop/bin/yarn --config ./conf jar share/hadoop/mapreduce/hadoop-mapreduce-examples-0.24.0-SNAPSHOT.jar randomwriter out
      Running 0 maps.
      Job started: Wed Mar 07 23:28:23 UTC 2012
      java.lang.IllegalArgumentException: java.net.UnknownHostException: ns1
      	at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:431)
      	at org.apache.hadoop.security.SecurityUtil.buildDTServiceName(SecurityUtil.java:312)
      	at org.apache.hadoop.fs.FileSystem.getCanonicalServiceName(FileSystem.java:217)
      	at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:119)
      	at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:97)
      	at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:80)
      	at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:137)
      	at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:411)
      	at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:326)
      	at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1221)
      	at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218)
      ....
      

      0.24

      1. HDFS-3062-trunk.patch
        2 kB
        Mingjie Lai
      2. HDFS-3062-trunk-2.patch
        2 kB
        Mingjie Lai

        Issue Links

          Activity

          No work has yet been logged on this issue.

            People

            • Assignee:
              Mingjie Lai
              Reporter:
              Mingjie Lai
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development