[FLINK-13753] Integrate new Source Operator with Mailbox Model in StreamTask

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Done
• Component/s: Runtime / Task
• Assignee: Zhijiang
• Reporter: Zhijiang
• Created: 16/Aug/19 17:19
• Updated: 02/Jul/20 06:30
• Resolved: 02/Jul/20 06:30

Description

This is the umbrella issue for integrating new source operator with mailbox model in StreamTask.

The motivation is based on FLIP-27 which proposes to refactor the whole source API and the integration of task-level actions (including checkpoint, timer, async operator) with unified mailbox model on runtime side.

• The benefits are simple unified processing logics because only one single thread handles all the actions without concurrent issue, and further getting rid of lock dependency which causes unfair lock concern in checkpoint process.
• We still need to support the current legacy source in some releases which would probably be used for a while, especially for the scenario of performance concern.

The design doc is https://docs.google.com/document/d/13x9M7k1SRqkOFXP0bETcJemIRyJzoqGgkdy11pz5qHM/edit#

Sub-Tasks

• Decouple OperatorChain with StreamStatusMaintainer: Closed
• Add the exception throwable in the interface methods of ValveOutputHandler: Closed
• Pass the counter of numRecordsIn into the constructor of StreamInputProcessor: Resolved
• Introduce TwoInputSelectionHandler for selecting input in StreamTwoInputSelectableProcessor: Closed
• Refactor the implementation of StreamInputProcessor based on StreamTaskInput#emitNext: Closed
• Refactor StreamInputProcessor#processInput based on InputStatus: Closed
• Refactor the process of checking stream status while emitting watermark in source: Closed
• Define SourceReaderOperator to verify the integration with StreamOneInputProcessor: Resolved