[FLINK-29617] Cost too much time to start SourceCoordinator of hdfsFileSource when start JobMaster

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.15.3
• Component/s: Connectors / FileSystem, Runtime / Coordination
• Labels: coordination file-system stale-assigned
• Assignee: luoyuxia
• Reporter: LI Mingkun
• Created: 13/Oct/22 07:01
• Updated: 15/Aug/23 10:35

Description

Scenario:

Our user use flink batch to compact small files in one day. Flink version : 1.15
He split pipeline into 24 for each hour. So there are 24 source
 
I find it  costs too much time to start SourceCoordinator of hdfsFileSource when start JobMaster
 
 as follow:
 

 

Root Cause:

I got the root cause after check: 

1. AbstractFileSource will enumerateSplits when createEnumerator
2. NotSplittingRecursiveEnumerator need to get fileblockLocation of every fileblock which is a heavy IO operation

 

 

Suggestion

1. FileSource add option to disable location fetcher
2. Move location fetcher into IOExecutor