[HADOOP-12563] Updated utility to create/modify token files

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha1
• Fix Version/s: 3.0.0-alpha1
• Hadoop Flags: Incompatible change, Reviewed
• Release Note: This feature introduces a new command called "hadoop dtutil" which lets users request and download delegation tokens with certain attributes.
• Assignee: Matthew Paduano
• Reporter: Allen Wittenauer
• Created: 10/Nov/15 06:09
• Updated: 18/Dec/19 18:26
• Resolved: 30/Apr/16 05:33

Description

hdfs fetchdt is missing some critical features and is geared almost exclusively towards HDFS operations. Additionally, the token files that are created use Java serializations which are hard/impossible to deal with in other languages. It should be replaced with a better utility in common that can read/write protobuf-based token files, has enough flexibility to be used with other services, and offers key functionality such as append and rename. The old version file format should still be supported for backward compatibility, but will be effectively deprecated.

A follow-on JIRA will deprecrate fetchdt.

Issue Links

• Blocked: YARN-4435 Add RM Delegation Token DtFetcher Implementation for DtUtil
• breaks: HADOOP-15059 3.0 deployment cannot work with old version MR tar ball which breaks rolling upgrade
• breaks: HADOOP-13049 Fix the TestFailures After HADOOP-12563
• duplicates: HADOOP-9837 Hadoop Token Command
• is depended upon by: HADOOP-12649 Improve Kerberos diagnostics and failure handling
• is related to: HADOOP-13123 Permit the default hadoop delegation token file format to be configurable
• relates to: HADOOP-16621 [pb-upgrade] Remove Protobuf classes from signatures of Public APIs
• relates to: SPARK-11265 YarnClient can't get tokens to talk to Hive 1.2.1 in a secure cluster
• relates to: SLIDER-1081 add a command, "slider tokens" to save/list tokens to a file
• relates to: HADOOP-15885 Add base64 (urlString) support to DTUtil
• relates to: SPARK-12241 Improve failure reporting in Yarn client obtainTokenForHBase()
• requires: HADOOP-13054 Use proto delimited IO to fix tests broken by HADOOP-12563