Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-6667

In HDFS HA mode, Distcp/SLive with webhdfs on secure cluster fails with Client cannot authenticate via:[TOKEN, KERBEROS] error

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.6.0
    • Component/s: security
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Opening on Arpit Gupta's behalf.

      We observed that, in HDFS HA mode, running Distcp/SLive with webhdfs will fail on YARN. In non-HA mode, it'll pass.

      The reason is in HA mode, only webhdfs delegation token is generated for the job, but YARN also requires the regular hdfs token to do localization, log-aggregation etc.
      In non-HA mode, both tokens are generated for the job.

        Activity

        Jian He created issue -
        Jian He made changes -
        Field Original Value New Value
        Description Opening on [~arpitgupta]'s behalf.

        We observed that, in HDFS HA mode, running Distcp/SLive with webhdfs will fail on YARN. In non-HA mode, it'll pass.

        The reason is in HA mode, only webhdfs delegation token is generated for the job, but YARN also requires the regular hdfs to do localization, log-aggregation etc.
        In non-HA mode, both tokens are generated for the job.
        Opening on [~arpitgupta]'s behalf.

        We observed that, in HDFS HA mode, running Distcp/SLive with webhdfs will fail on YARN. In non-HA mode, it'll pass.

        The reason is in HA mode, only webhdfs delegation token is generated for the job, but YARN also requires the regular hdfs token to do localization, log-aggregation etc.
        In non-HA mode, both tokens are generated for the job.
        Tsz Wo Nicholas Sze made changes -
        Component/s security [ 12312526 ]
        Jing Zhao made changes -
        Project Hadoop Common [ 12310240 ] Hadoop HDFS [ 12310942 ]
        Key HADOOP-10519 HDFS-6667
        Component/s security [ 12313400 ]
        Component/s security [ 12312526 ]
        Jing Zhao made changes -
        Attachment HDFS-6667.000.patch [ 12655621 ]
        Jing Zhao made changes -
        Assignee Jing Zhao [ jingzhao ]
        Jing Zhao made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Jing Zhao made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags Reviewed [ 10343 ]
        Fix Version/s 2.6.0 [ 12327181 ]
        Resolution Fixed [ 1 ]
        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Jing Zhao
            Reporter:
            Jian He
          • Votes:
            0 Vote for this issue
            Watchers:
            12 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development