[FLINK-36540] Add Support for Hadoop Caller Context when using Flink on yarn

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Runtime / Task
• Labels: pull-request-available

Environment

Hadoop 3.3.4
Flink 1.14.4

• Assignee: Unassigned
• Reporter: liang yu
• Created: 15/Oct/24 08:29
• Updated: 06/Nov/24 13:52

Description

Description: As HDFS mentioned in HDFS-9184, for a given HDFS operation, it's
very helpful to track which upper level job issues it. But now Flink does not support to set callerContext when trying to operate on Hadoop fileSystem.

 

implementation: Add a new HadoopOption which can be set in flink.yaml. By setting this option to true, flink can search for the classpath if we can invoke the method

callerContext.Builder().build().setCurrent(context)

 and then set the jobId and TaskName with subtask id so that we can find exactly who did the operation to Hadoop file.

Issue Links

• is duplicated by: FLINK-36539 Add Support for Hadoop Caller Context when using Flink on yarn
• links to: GitHub Pull Request #25516