[GOBBLIN-1286] Support fetching multiple DFS tokens for HDFS Federation

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.15.0
• Fix Version/s: 0.17.0
• Component/s: misc
• Assignee: Unassigned
• Reporter: Sandeep Devaiah
• Created: 13/Oct/20 14:04
• Updated: 14/Jun/23 04:13

Description

Currently org.apache.gobblin.util.hadoop.TokenUtils#getHdfsToken adds the DistributedFileSystem's delegation token to the Credential's object. However, if the DFS is federated, then only one token for the primary name node is fetched and tokens for child namenodes are not fetched.

Hence this method needs to be updated to use org.apache.hadoop.fs.FileSystem#addDelegationTokens instead of org.apache.hadoop.fs.FileSystem#getDelegationTokens which automatically fetches the delegation tokens for all child FS as well.

Issue Links

• links to: https://github.com/apache/incubator-gobblin/pull/3125