[YARN-10333] YarnClient obtain Delegation Token for Log Aggregation Path

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.4.0
• Component/s: log-aggregation
• Hadoop Flags: Reviewed
• Assignee: Prabhu Joseph
• Reporter: Prabhu Joseph
• Created: 30/Jun/20 19:02
• Updated: 29/Jan/21 23:29
• Resolved: 09/Jul/20 07:20

Description

There are use cases where Yarn Log Aggregation Path is configured to a FileSystem like S3 or ABFS different from what is configured in fs.defaultFS (HDFS). Log Aggregation fails as the client has token only for fs.defaultFS and not for log aggregation path.

This Jira is to improve YarnClient by obtaining delegation token for log aggregation path and add it to the Credential of Container Launch Context similar to how it does for Timeline Delegation Token.

Issue Links

• causes: YARN-10601 The Yarn client should use the UGI who created the Yarn client for obtaining a delegation token for the remote log dir