Details
-
Sub-task
-
Status: Closed
-
Major
-
Resolution: Implemented
-
1.6.1, 1.6.2, 1.7.0
Description
Today DataSource Task pull InputSplits from JobManager to achieve better performance, however, when a DataSourceTask failed and rerun, it will not get the same splits as its previous version. this will introduce inconsistent result or even data corruption.
Furthermore, if there are two executions run at the same time (in batch scenario), this two executions should process same splits.
we need to fix the issue to make the inputs of a DataSourceTask deterministic. The propose is save all splits into ExecutionVertex and DataSourceTask will pull split from there.
document:
https://docs.google.com/document/d/1FdZdcA63tPUEewcCimTFy9Iz2jlVlMRANZkO4RngIuk/edit?usp=sharing
Attachments
Issue Links
- is duplicated by
-
FLINK-8442 Should recovery the input split when an execution failover with FailoverRegion
- Closed
- links to