[YARN-5305] Yarn Application Log Aggregation fails due to NM can not get correct HDFS delegation token III

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.5.0
• Component/s: yarn
• Labels: pull-request-available
• Hadoop Flags: Reviewed
• Assignee: Peter Szucs
• Reporter: Xianyin Xin
• Created: 01/Jul/16 07:13
• Updated: 20/Mar/24 14:34
• Resolved: 20/Mar/24 14:34

Description

Different with YARN-5098 and YARN-5302, this problem happens when AM submits a startContainer request with a new HDFS token (say, tokenB) which is not managed by YARN, so two tokens exist in the credentials of the user on NM, one is tokenB, the other is the one renewed on RM (tokenA). If tokenB is selected when connect to HDFS and tokenB expires, exception happens.

Supplementary: this problem happen due to that AM didn't use the service name as the token alias in credentials, so two tokens for the same service can co-exist in one credentials. TokenSelector can only select the first matched token, it doesn't care if the token is valid or not.

Issue Links

• is related to: YARN-5175 Simplify the delegation token renewal management for YARN localization and log-aggregation
• relates to: YARN-5302 Yarn Application log Aggregation fails due to NM can not get correct HDFS delegation token II
• links to: GitHub Pull Request #6625