Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-6667

In HDFS HA mode, Distcp/SLive with webhdfs on secure cluster fails with Client cannot authenticate via:[TOKEN, KERBEROS] error

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.6.0
    • Component/s: security
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Opening on Arpit Gupta's behalf.

      We observed that, in HDFS HA mode, running Distcp/SLive with webhdfs will fail on YARN. In non-HA mode, it'll pass.

      The reason is in HA mode, only webhdfs delegation token is generated for the job, but YARN also requires the regular hdfs token to do localization, log-aggregation etc.
      In non-HA mode, both tokens are generated for the job.

        Activity

        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        87d 20h 4m 1 Jing Zhao 14/Jul/14 22:18
        Patch Available Patch Available Resolved Resolved
        3d 1h 57m 1 Jing Zhao 18/Jul/14 00:16
        Resolved Resolved Closed Closed
        136d 3h 51m 1 Arun C Murthy 01/Dec/14 03:08
        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Hide
        Jing Zhao added a comment -

        Sure. I just followed your suggestion and change the service for HA webhdfs tokens to "ha-webhdfs:logicalURI". For HA hdfs tokens, the service name remains to be "ha-hdfs:logicalURI".

        I've tested the distcp using hdfs and webhdfs (with HA and non-HA filesystem URI) in a secured cluster. Since we have not changed the code paths in 1) hdfs + non-HA, 2) hdfs + HA, and 3) webhdfs + non-HA, and webhdfs+HA did not work before the change, I think we will not cause any regression here. But we will keep testing all the scenarios including running jobs during a rolling upgrade.

        Show
        Jing Zhao added a comment - Sure. I just followed your suggestion and change the service for HA webhdfs tokens to "ha-webhdfs:logicalURI". For HA hdfs tokens, the service name remains to be "ha-hdfs:logicalURI". I've tested the distcp using hdfs and webhdfs (with HA and non-HA filesystem URI) in a secured cluster. Since we have not changed the code paths in 1) hdfs + non-HA, 2) hdfs + HA, and 3) webhdfs + non-HA, and webhdfs+HA did not work before the change, I think we will not cause any regression here. But we will keep testing all the scenarios including running jobs during a rolling upgrade.
        Hide
        Daryn Sharp added a comment -

        Jing Zhao, I'm glancing at the patch. My time is short right now, so to accelerate my post-review, could you please:

        1. Explain the mechanics of the change you made
        2. Confirm that no incompatibilities have been introduced that will break jobs during a rolling upgrade
        Show
        Daryn Sharp added a comment - Jing Zhao , I'm glancing at the patch. My time is short right now, so to accelerate my post-review, could you please: Explain the mechanics of the change you made Confirm that no incompatibilities have been introduced that will break jobs during a rolling upgrade
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #1808 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1808/)
        HDFS-6667. In HDFS HA mode, Distcp/SLive with webhdfs on secure cluster fails with Client cannot authenticate via:[TOKEN, KERBEROS] error. Contributed by Jing Zhao. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1611508)

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HAUtil.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/NameNodeProxies.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/DatanodeWebHdfsMethods.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDelegationTokensWithHA.java
        Show
        Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #1808 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1808/ ) HDFS-6667 . In HDFS HA mode, Distcp/SLive with webhdfs on secure cluster fails with Client cannot authenticate via: [TOKEN, KERBEROS] error. Contributed by Jing Zhao. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1611508 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HAUtil.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/NameNodeProxies.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/DatanodeWebHdfsMethods.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDelegationTokensWithHA.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #1835 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1835/)
        HDFS-6667. In HDFS HA mode, Distcp/SLive with webhdfs on secure cluster fails with Client cannot authenticate via:[TOKEN, KERBEROS] error. Contributed by Jing Zhao. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1611508)

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HAUtil.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/NameNodeProxies.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/DatanodeWebHdfsMethods.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDelegationTokensWithHA.java
        Show
        Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1835 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1835/ ) HDFS-6667 . In HDFS HA mode, Distcp/SLive with webhdfs on secure cluster fails with Client cannot authenticate via: [TOKEN, KERBEROS] error. Contributed by Jing Zhao. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1611508 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HAUtil.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/NameNodeProxies.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/DatanodeWebHdfsMethods.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDelegationTokensWithHA.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #616 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/616/)
        HDFS-6667. In HDFS HA mode, Distcp/SLive with webhdfs on secure cluster fails with Client cannot authenticate via:[TOKEN, KERBEROS] error. Contributed by Jing Zhao. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1611508)

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HAUtil.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/NameNodeProxies.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/DatanodeWebHdfsMethods.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDelegationTokensWithHA.java
        Show
        Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #616 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/616/ ) HDFS-6667 . In HDFS HA mode, Distcp/SLive with webhdfs on secure cluster fails with Client cannot authenticate via: [TOKEN, KERBEROS] error. Contributed by Jing Zhao. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1611508 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HAUtil.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/NameNodeProxies.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/DatanodeWebHdfsMethods.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDelegationTokensWithHA.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #5905 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5905/)
        HDFS-6667. In HDFS HA mode, Distcp/SLive with webhdfs on secure cluster fails with Client cannot authenticate via:[TOKEN, KERBEROS] error. Contributed by Jing Zhao. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1611508)

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HAUtil.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/NameNodeProxies.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/DatanodeWebHdfsMethods.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDelegationTokensWithHA.java
        Show
        Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #5905 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5905/ ) HDFS-6667 . In HDFS HA mode, Distcp/SLive with webhdfs on secure cluster fails with Client cannot authenticate via: [TOKEN, KERBEROS] error. Contributed by Jing Zhao. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1611508 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HAUtil.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/NameNodeProxies.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/DatanodeWebHdfsMethods.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDelegationTokensWithHA.java
        Jing Zhao made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags Reviewed [ 10343 ]
        Fix Version/s 2.6.0 [ 12327181 ]
        Resolution Fixed [ 1 ]
        Hide
        Jing Zhao added a comment -

        I've committed this to trunk and branch-2.

        Show
        Jing Zhao added a comment - I've committed this to trunk and branch-2.
        Hide
        Jing Zhao added a comment -

        I will commit the patch based on Haohui Mai's +1. We can address further comments in separate jiras.

        Show
        Jing Zhao added a comment - I will commit the patch based on Haohui Mai 's +1. We can address further comments in separate jiras.
        Hide
        Haohui Mai added a comment -

        Looks good to me. +1. I think that this patch implements the approach proposed by Daryn.

        Daryn Sharp, do you have any comments?

        Show
        Haohui Mai added a comment - Looks good to me. +1. I think that this patch implements the approach proposed by Daryn. Daryn Sharp , do you have any comments?
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12655621/HDFS-6667.000.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.hdfs.tools.TestDFSAdminWithHA
        org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7347//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7347//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12655621/HDFS-6667.000.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.tools.TestDFSAdminWithHA org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7347//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7347//console This message is automatically generated.
        Hide
        Jing Zhao added a comment -

        The unit test failures should be unrelated. TestDFSAdminWithHA and TestPipelinesFailover were also seen in recent Jenkins run such as here. TestProcessCorruptBlocks has been reported in HDFS-6656.

        Show
        Jing Zhao added a comment - The unit test failures should be unrelated. TestDFSAdminWithHA and TestPipelinesFailover were also seen in recent Jenkins run such as here . TestProcessCorruptBlocks has been reported in HDFS-6656 .
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12655621/HDFS-6667.000.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.hdfs.tools.TestDFSAdminWithHA
        org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
        org.apache.hadoop.hdfs.server.namenode.TestProcessCorruptBlocks

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7344//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7344//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12655621/HDFS-6667.000.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.tools.TestDFSAdminWithHA org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover org.apache.hadoop.hdfs.server.namenode.TestProcessCorruptBlocks +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7344//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7344//console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12655621/HDFS-6667.000.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.hdfs.tools.TestDFSAdminWithHA
        org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7341//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7341//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12655621/HDFS-6667.000.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.tools.TestDFSAdminWithHA org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7341//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7341//console This message is automatically generated.
        Jing Zhao made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Jing Zhao made changes -
        Assignee Jing Zhao [ jingzhao ]
        Jing Zhao made changes -
        Attachment HDFS-6667.000.patch [ 12655621 ]
        Hide
        Jing Zhao added a comment -

        Upload a simple patch based on Daryn Sharp's suggestion. I've tested it in a secured cluster and distcp (using HA webhdfs) works with the patch.

        Show
        Jing Zhao added a comment - Upload a simple patch based on Daryn Sharp 's suggestion. I've tested it in a secured cluster and distcp (using HA webhdfs) works with the patch.
        Jing Zhao made changes -
        Project Hadoop Common [ 12310240 ] Hadoop HDFS [ 12310942 ]
        Key HADOOP-10519 HDFS-6667
        Component/s security [ 12313400 ]
        Component/s security [ 12312526 ]
        Hide
        Daryn Sharp added a comment -

        I never liked the way hdfs tokens are managed. There is no difference between a hdfs, (s)webhdfs, hftp, etc token so the token kind should be the same. Unfortunately the service field represents the issuer's address for renewal, as well as the key for token selection for connections so token duping hacks are currently used. I've always meant to move to servers returning an opaque server-id for token selection that would make the protocol irrelevant... For HA servers, the opaque server-id would be the HA logical name so the same token would work with both hdfs and webhdfs. But I digress.

        All that said, the short answer for now is the service for logical HA webhdfs tokens should be "ha-webhdfs:hostname".

        Show
        Daryn Sharp added a comment - I never liked the way hdfs tokens are managed. There is no difference between a hdfs, (s)webhdfs, hftp, etc token so the token kind should be the same. Unfortunately the service field represents the issuer's address for renewal, as well as the key for token selection for connections so token duping hacks are currently used. I've always meant to move to servers returning an opaque server-id for token selection that would make the protocol irrelevant... For HA servers, the opaque server-id would be the HA logical name so the same token would work with both hdfs and webhdfs. But I digress. All that said, the short answer for now is the service for logical HA webhdfs tokens should be "ha-webhdfs:hostname".
        Tsz Wo Nicholas Sze made changes -
        Component/s security [ 12312526 ]
        Hide
        Jian He added a comment -
        Show
        Jian He added a comment - Daryn Sharp , Jing Zhao , Haohui Mai , any ideas?
        Jian He made changes -
        Field Original Value New Value
        Description Opening on [~arpitgupta]'s behalf.

        We observed that, in HDFS HA mode, running Distcp/SLive with webhdfs will fail on YARN. In non-HA mode, it'll pass.

        The reason is in HA mode, only webhdfs delegation token is generated for the job, but YARN also requires the regular hdfs to do localization, log-aggregation etc.
        In non-HA mode, both tokens are generated for the job.
        Opening on [~arpitgupta]'s behalf.

        We observed that, in HDFS HA mode, running Distcp/SLive with webhdfs will fail on YARN. In non-HA mode, it'll pass.

        The reason is in HA mode, only webhdfs delegation token is generated for the job, but YARN also requires the regular hdfs token to do localization, log-aggregation etc.
        In non-HA mode, both tokens are generated for the job.
        Hide
        Jian He added a comment -

        The following function seems some problem.
        In HA mode, getCanonicalServiceName() returns the HA-id (e.g. ha-hdfs:hostname). Seems both webhdfs token and hdfs token share the same service name (ha-id), if the webhdfs token is created first, even if we are issuing a new request for the regular hdfs token, "token = credentials.getToken(service);" returns non-null (the webHdfsToken), and won't call getDelegationToken(renewer); to ask for the new regular hdfs token.

        private void collectDelegationTokens(final String renewer,
                                               final Credentials credentials,
                                               final List<Token<?>> tokens)
                                                   throws IOException {
            final String serviceName = getCanonicalServiceName();
            // Collect token of the this filesystem and then of its embedded children
            if (serviceName != null) { // fs has token, grab it
              final Text service = new Text(serviceName);
              LOG.info("serviceName " + serviceName);
              Token<?> token = credentials.getToken(service);
              if (token == null) {
                LOG.info("renewer " + renewer);
                token = getDelegationToken(renewer);
                if (token != null) {
                  tokens.add(token);
                  credentials.addToken(service, token);
                }
              }
            }
            // Now collect the tokens from the children
            final FileSystem[] children = getChildFileSystems();
            if (children != null) {
              for (final FileSystem fs : children) {
                fs.collectDelegationTokens(renewer, credentials, tokens);
              }
            }
          }
        
        Show
        Jian He added a comment - The following function seems some problem. In HA mode, getCanonicalServiceName() returns the HA-id (e.g. ha-hdfs:hostname). Seems both webhdfs token and hdfs token share the same service name (ha-id), if the webhdfs token is created first, even if we are issuing a new request for the regular hdfs token, "token = credentials.getToken(service);" returns non-null (the webHdfsToken), and won't call getDelegationToken(renewer); to ask for the new regular hdfs token. private void collectDelegationTokens( final String renewer, final Credentials credentials, final List<Token<?>> tokens) throws IOException { final String serviceName = getCanonicalServiceName(); // Collect token of the this filesystem and then of its embedded children if (serviceName != null ) { // fs has token, grab it final Text service = new Text(serviceName); LOG.info( "serviceName " + serviceName); Token<?> token = credentials.getToken(service); if (token == null ) { LOG.info( "renewer " + renewer); token = getDelegationToken(renewer); if (token != null ) { tokens.add(token); credentials.addToken(service, token); } } } // Now collect the tokens from the children final FileSystem[] children = getChildFileSystems(); if (children != null ) { for ( final FileSystem fs : children) { fs.collectDelegationTokens(renewer, credentials, tokens); } } }
        Hide
        Jian He added a comment -

        Thanks Arpit Gupta for reporting this issue.

        Show
        Jian He added a comment - Thanks Arpit Gupta for reporting this issue.
        Jian He created issue -

          People

          • Assignee:
            Jing Zhao
            Reporter:
            Jian He
          • Votes:
            0 Vote for this issue
            Watchers:
            12 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development