[FLINK-10205] Batch Job: InputSplit Fault tolerant for DataSourceTask

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Implemented
• Affects Version/s: 1.6.1, 1.6.2, 1.7.0
• Fix Version/s: 1.9.0
• Component/s: Runtime / Coordination
• Labels: pull-request-available
• Assignee: ryantaocer
• Reporter: JIN SUN
• Created: 24/Aug/18 02:19
• Updated: 01/Jun/21 03:40
• Resolved: 10/Apr/19 09:46

Description

Today DataSource Task pull InputSplits from JobManager to achieve better performance, however, when a DataSourceTask failed and rerun, it will not get the same splits as its previous version. this will introduce inconsistent result or even data corruption.

Furthermore,  if there are two executions run at the same time (in batch scenario), this two executions should process same splits.

we need to fix the issue to make the inputs of a DataSourceTask deterministic. The propose is save all splits into ExecutionVertex and DataSourceTask will pull split from there.

 document:

https://docs.google.com/document/d/1FdZdcA63tPUEewcCimTFy9Iz2jlVlMRANZkO4RngIuk/edit?usp=sharing

Issue Links

• is duplicated by: FLINK-8442 Should recovery the input split when an execution failover with FailoverRegion
• links to: GitHub Pull Request #6657
• links to: GitHub Pull Request #6684
• links to: GitHub Pull Request #8125