[FLINK-19568] Offload creating TM launch contexts to the IO executor

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Done
• Fix Version/s: 1.12.0
• Component/s: Deployment / YARN
• Labels: pull-request-available
• Assignee: Xintong Song
• Reporter: Xintong Song
• Created: 12/Oct/20 06:39
• Updated: 13/Oct/20 01:47
• Resolved: 13/Oct/20 01:47

Description

Currently, for launching each TM container on Yarn, Flink creates a container launch context in RM's PRC main thread. This includes accessing file status from remote file systems, which may blocks the RM's main thread, especially when remote file system is slow. See this thread.

The creating of TM context does not access nor change any RM's internal states. Therefore, I propose to offload the work to the IO executor. To be specific, I think the entire YarnResourceManagerDriver#createTaskExecutorLaunchContext can be invoked on the IO executor.

Issue Links

• links to: GitHub Pull Request #13591