Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-3062

Fail to submit mapred job on a secured-HA-HDFS: logic URI cannot be picked up by job submission.

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.24.0
    • Fix Version/s: 2.0.0-alpha
    • Component/s: ha, security
    • Labels:
      None

      Description

      When testing the combination of NN HA + security + yarn, I found that the mapred job submission cannot pick up the logic URI of a nameservice.

      I have logic URI configured in core-site.xml

      <property>
       <name>fs.defaultFS</name>
       <value>hdfs://ns1</value>
      </property>
      

      HDFS client can work with the HA deployment/configs:

      [root@nn1 hadoop]# hdfs dfs -ls /
      Found 6 items
      drwxr-xr-x   - hbase  hadoop          0 2012-03-07 20:42 /hbase
      drwxrwxrwx   - yarn   hadoop          0 2012-03-07 20:42 /logs
      drwxr-xr-x   - mapred hadoop          0 2012-03-07 20:42 /mapred
      drwxr-xr-x   - mapred hadoop          0 2012-03-07 20:42 /mr-history
      drwxrwxrwt   - hdfs   hadoop          0 2012-03-07 21:57 /tmp
      drwxr-xr-x   - hdfs   hadoop          0 2012-03-07 20:42 /user
      

      but cannot submit a mapred job with security turned on

      [root@nn1 hadoop]# /usr/lib/hadoop/bin/yarn --config ./conf jar share/hadoop/mapreduce/hadoop-mapreduce-examples-0.24.0-SNAPSHOT.jar randomwriter out
      Running 0 maps.
      Job started: Wed Mar 07 23:28:23 UTC 2012
      java.lang.IllegalArgumentException: java.net.UnknownHostException: ns1
      	at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:431)
      	at org.apache.hadoop.security.SecurityUtil.buildDTServiceName(SecurityUtil.java:312)
      	at org.apache.hadoop.fs.FileSystem.getCanonicalServiceName(FileSystem.java:217)
      	at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:119)
      	at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:97)
      	at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:80)
      	at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:137)
      	at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:411)
      	at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:326)
      	at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1221)
      	at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218)
      ....
      

      0.24

      1. HDFS-3062-trunk.patch
        2 kB
        Mingjie Lai
      2. HDFS-3062-trunk-2.patch
        2 kB
        Mingjie Lai

        Issue Links

          Activity

          Hide
          Matt Foley added a comment -

          Commit r1401158 | suresh | 2012-10-22 21:34:49 -0700 (Mon, 22 Oct 2012)
          incorrectly attributed this bug to a fix in release 1.2.0, in both CHANGES.txt and svn log.
          The correct reference should have been HDFS-4062 not 3062. Fixed CHANGES.txt.

          Show
          Matt Foley added a comment - Commit r1401158 | suresh | 2012-10-22 21:34:49 -0700 (Mon, 22 Oct 2012) incorrectly attributed this bug to a fix in release 1.2.0, in both CHANGES.txt and svn log. The correct reference should have been HDFS-4062 not 3062. Fixed CHANGES.txt.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1021 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1021/)
          HDFS-3062. Fix bug which prevented MR job submission from creating delegation tokens on an HA cluster. Contributed by Mingjie Lai. (Revision 1301285)

          Result = FAILURE
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301285
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDelegationTokensWithHA.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1021 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1021/ ) HDFS-3062 . Fix bug which prevented MR job submission from creating delegation tokens on an HA cluster. Contributed by Mingjie Lai. (Revision 1301285) Result = FAILURE todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301285 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDelegationTokensWithHA.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Build #227 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/227/)
          HDFS-3062. Fix bug which prevented MR job submission from creating delegation tokens on an HA cluster. Contributed by Mingjie Lai. (Revision 1301286)

          Result = FAILURE
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301286
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDelegationTokensWithHA.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #227 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/227/ ) HDFS-3062 . Fix bug which prevented MR job submission from creating delegation tokens on an HA cluster. Contributed by Mingjie Lai. (Revision 1301286) Result = FAILURE todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301286 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDelegationTokensWithHA.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #199 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/199/)
          HDFS-3062. Fix bug which prevented MR job submission from creating delegation tokens on an HA cluster. Contributed by Mingjie Lai. (Revision 1301286)

          Result = SUCCESS
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301286
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDelegationTokensWithHA.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #199 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/199/ ) HDFS-3062 . Fix bug which prevented MR job submission from creating delegation tokens on an HA cluster. Contributed by Mingjie Lai. (Revision 1301286) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301286 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDelegationTokensWithHA.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #986 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/986/)
          HDFS-3062. Fix bug which prevented MR job submission from creating delegation tokens on an HA cluster. Contributed by Mingjie Lai. (Revision 1301285)

          Result = SUCCESS
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301285
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDelegationTokensWithHA.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #986 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/986/ ) HDFS-3062 . Fix bug which prevented MR job submission from creating delegation tokens on an HA cluster. Contributed by Mingjie Lai. (Revision 1301285) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301285 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDelegationTokensWithHA.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #1892 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1892/)
          HDFS-3062. Fix bug which prevented MR job submission from creating delegation tokens on an HA cluster. Contributed by Mingjie Lai. (Revision 1301285)

          Result = ABORTED
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301285
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDelegationTokensWithHA.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1892 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1892/ ) HDFS-3062 . Fix bug which prevented MR job submission from creating delegation tokens on an HA cluster. Contributed by Mingjie Lai. (Revision 1301285) Result = ABORTED todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301285 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDelegationTokensWithHA.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Commit #695 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/695/)
          HDFS-3062. Fix bug which prevented MR job submission from creating delegation tokens on an HA cluster. Contributed by Mingjie Lai. (Revision 1301286)

          Result = ABORTED
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301286
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDelegationTokensWithHA.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #695 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/695/ ) HDFS-3062 . Fix bug which prevented MR job submission from creating delegation tokens on an HA cluster. Contributed by Mingjie Lai. (Revision 1301286) Result = ABORTED todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301286 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDelegationTokensWithHA.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #1958 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1958/)
          HDFS-3062. Fix bug which prevented MR job submission from creating delegation tokens on an HA cluster. Contributed by Mingjie Lai. (Revision 1301285)

          Result = SUCCESS
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301285
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDelegationTokensWithHA.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1958 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1958/ ) HDFS-3062 . Fix bug which prevented MR job submission from creating delegation tokens on an HA cluster. Contributed by Mingjie Lai. (Revision 1301285) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301285 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDelegationTokensWithHA.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Commit #678 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/678/)
          HDFS-3062. Fix bug which prevented MR job submission from creating delegation tokens on an HA cluster. Contributed by Mingjie Lai. (Revision 1301286)

          Result = SUCCESS
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301286
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDelegationTokensWithHA.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #678 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/678/ ) HDFS-3062 . Fix bug which prevented MR job submission from creating delegation tokens on an HA cluster. Contributed by Mingjie Lai. (Revision 1301286) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301286 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDelegationTokensWithHA.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #1883 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1883/)
          HDFS-3062. Fix bug which prevented MR job submission from creating delegation tokens on an HA cluster. Contributed by Mingjie Lai. (Revision 1301285)

          Result = SUCCESS
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301285
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDelegationTokensWithHA.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1883 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1883/ ) HDFS-3062 . Fix bug which prevented MR job submission from creating delegation tokens on an HA cluster. Contributed by Mingjie Lai. (Revision 1301285) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301285 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDelegationTokensWithHA.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-0.23-Commit #687 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/687/)
          HDFS-3062. Fix bug which prevented MR job submission from creating delegation tokens on an HA cluster. Contributed by Mingjie Lai. (Revision 1301286)

          Result = SUCCESS
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301286
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDelegationTokensWithHA.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #687 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/687/ ) HDFS-3062 . Fix bug which prevented MR job submission from creating delegation tokens on an HA cluster. Contributed by Mingjie Lai. (Revision 1301286) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301286 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDelegationTokensWithHA.java
          Hide
          Todd Lipcon added a comment -

          Committed to branch-23 and trunk, thanks Mingjie

          Show
          Todd Lipcon added a comment - Committed to branch-23 and trunk, thanks Mingjie
          Hide
          Todd Lipcon added a comment -

          +1, will commit momentarily. Thanks for fixing this, Mingjie.

          Show
          Todd Lipcon added a comment - +1, will commit momentarily. Thanks for fixing this, Mingjie.
          Hide
          Mingjie Lai added a comment -

          The test error is reported at HDFS-3106. It's not caused by the patch here.

          Show
          Mingjie Lai added a comment - The test error is reported at HDFS-3106 . It's not caused by the patch here.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12517984/HDFS-3062-trunk-2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.cli.TestHDFSCLI

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2020//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2020//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12517984/HDFS-3062-trunk-2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.cli.TestHDFSCLI +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2020//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2020//console This message is automatically generated.
          Hide
          Mingjie Lai added a comment -

          New patch addresses Todd's comments.

          Show
          Mingjie Lai added a comment - New patch addresses Todd's comments.
          Hide
          Todd Lipcon added a comment -

          Thanks for the patch, Mingjie! A small formatting nit: can you change the // comments for the new test case to be a javadoc-style comment, and move the @Test annotation to be on the line just above the method? This way it will match the style used elsewhere.

          Show
          Todd Lipcon added a comment - Thanks for the patch, Mingjie! A small formatting nit: can you change the // comments for the new test case to be a javadoc-style comment, and move the @Test annotation to be on the line just above the method? This way it will match the style used elsewhere.
          Hide
          Mingjie Lai added a comment -

          A patch for the issue: overriding getCanonicalServiceName() at DistributedFileSystem.

          After this patch gets applied, I can have a mapred job running with ha + security.

          Show
          Mingjie Lai added a comment - A patch for the issue: overriding getCanonicalServiceName() at DistributedFileSystem. After this patch gets applied, I can have a mapred job running with ha + security.
          Hide
          Todd Lipcon added a comment -

          Looks like we need to override FileSystem.getCanonicalServiceName in DistributedFileSystem so that the canonical name is just the logical name, for the case of HA HDFS file systems.

          Show
          Todd Lipcon added a comment - Looks like we need to override FileSystem.getCanonicalServiceName in DistributedFileSystem so that the canonical name is just the logical name, for the case of HA HDFS file systems.
          Hide
          Mingjie Lai added a comment -

          It really weird that FileSystem.getCanonicalServiceName() is calling SecurityUtil.buildDTServiceName() to get the canonical service name. SecurityUtil.buildDTServiceName() tries to create a socket by the passed URI. Since it is a logic URI, it fails the job submission.

          Show
          Mingjie Lai added a comment - It really weird that FileSystem.getCanonicalServiceName() is calling SecurityUtil.buildDTServiceName() to get the canonical service name. SecurityUtil.buildDTServiceName() tries to create a socket by the passed URI. Since it is a logic URI, it fails the job submission.

            People

            • Assignee:
              Mingjie Lai
              Reporter:
              Mingjie Lai
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development