[MAPREDUCE-2764] Fix renewal of dfs delegation tokens

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.20.205.0, 0.23.0
• Release Note: Generalizes token renewal and canceling to a common interface and provides a plugin interface for adding renewers for new kinds of tokens. Hftp changed to store the tokens as HFTP and renew them over http.
• Assignee: Owen O'Malley
• Reporter: Daryn Sharp
• Created: 02/Aug/11 01:13
• Updated: 09/Mar/15 20:16
• Resolved: 14/Oct/11 06:20

Description

The JT may have issues renewing hftp tokens which disrupt long distcp jobs. The problem is the JT's delegation token renewal code is built on brittle assumptions. The token's service field contains only the "ip:port" pair. The renewal process assumes that the scheme must be hdfs. If that fails due to a VersionMismatchException, it tries https based on another assumption that it must be hftp if it's not hdfs. A number of other exceptions, most commonly IOExceptions, can be generated which fouls up the renewal since it won't fallback to https.

Issue Links

• breaks: HDFS-2326 hftp token renewal uses wrong port
• breaks: HDFS-2473 Fetching tokens from insecure cluster throws NPEs
• breaks: HDFS-2328 hftp throws NPE if security is not enabled on remote cluster
• incorporates: MAPREDUCE-2869 DelegationTokenRenewal should handle abstract delegation tokens
• incorporates: HDFS-2257 HftpFileSystem should implement getDelegationTokens
• incorporates: HADOOP-7644 Fix the delegation token tests to use the new style renewers
• incorporates: HADOOP-7625 TestDelegationToken is failing in 205
• is depended upon by: HDFS-2385 Support delegation token renewal in webhdfs
• is related to: MAPREDUCE-3192 Fix Javadoc warning in JobClient.java and Cluster.java
• relates to: HADOOP-7510 Tokens should use original hostname provided instead of ip
• requires: HADOOP-7541 Add issuer field to delegation tokens

Sub-Tasks

• Refactor token selectors to leverage common code: Open
• Standardize the value of token service: Resolved
• DelegationTokenRenewal should handle abstract delegation tokens: Closed