[FLINK-19069] finalizeOnMaster takes too much time and client timeouts

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Won't Fix
• Affects Version/s: 1.9.0, 1.10.0, 1.11.0, 1.12.0
• Component/s: Runtime / Coordination
• Labels: pull-request-available
• Assignee: Jiayi Liao
• Reporter: Jiayi Liao
• Created: 28/Aug/20 02:43
• Updated: 05/Oct/20 12:49
• Resolved: 05/Oct/20 12:49

Description

Currently we execute finalizeOnMaster in JM's main thread, which may stuck the JM for a very long time and client timeouts eventually.

For example, we'd like to write data to HDFS and commit files on JM, which takes more than ten minutes to commit tens of thousands files.

Issue Links

• links to: GitHub Pull Request #13412