[FLINK-10740] FLIP-27: Refactor Source Interface

• Type: Improvement
• Status: Open
• Priority: Not a Priority
• Resolution: Unresolved
• Component/s: API / DataSet, API / DataStream
• Labels: auto-deprioritized-major auto-deprioritized-minor auto-unassigned pull-request-available usability
• Assignee: Unassigned
• Reporter: Aljoscha Krettek
• Created: 31/Oct/18 16:16
• Updated: 16/Jul/22 10:38

Description

Please see the FLIP for any details: https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface

Issue Links

• blocks: FLINK-7286 Flink Dashboard fails to display bytes/records received by sources / emitted by sinks
• is related to: FLINK-22668 FLIP-150: Introduce Hybrid Source
• relates to: FLINK-21133 FLIP-27 Source does not work with synchronous savepoint
• links to: GitHub Pull Request #12492
• links to: GitHub Pull Request #13344
• mentioned in: Page Loading...

Sub-Tasks

• Implement OperatorEvent passing RPC.: Closed
• Add the interface and base implementation for SourceReader.: Closed
• Add implementation for SourceCoordinator: Closed
• Allow users to construct a DataStream from a new Source API: Closed
• Add Source API classes: Closed
• Integrate Operator Coordinators with Checkpoints: Closed
• Prerequisite cleanups and refactorings in the Checkpoint Coordinator: Closed
• Extend Checkpoint Coordinator to differentiate between "regional restore" and "full restore".: Closed
• Implement SourceReaderOperator which runs the SourceReader.: Closed
• Improve the performance of the RecordsWithSplitIds queue.: Closed
• Add common metrics to the SourceReader base implementation.: Open
• Add metrics to the source coordinator.: Open
• Enhance the OperatorEvent handling guarantee during checkpointing.: Closed
• Improve the `FutureCompletingBlockingQueue` to wakeup blocking put() more elegantly.: Closed
• Update the interfaces to PublicEvolving and add documentation.: Closed
• OperatorCoordinators receive failure notifications on task failure instead of on task restarts: Closed
• Add failure reason to OperatorCoordinator.failTask(...): Closed
• Simplify SourceOperator by using a utility SimpleVersionedListState: Closed
• Reduce scope for SourceOperator arguments and initialize more eagerly: Closed
• OperatorCoordinators must be notified of tasks cancelled as part of failover: Closed
• FutureCompletingBlockingQueue should have a capacity limit.: Closed
• OperatorCoordinator Context must support calls from thread other than JobMaster Main Thread: Closed
• Resolve stability annotations discussion for FLIP-27 in 1.11: Closed
• Remove Exceptions from signatures of SourceOutput methods: Closed
• Integrate FLIP-126 Timestamps and Watermarking with FLIP-27 sources: Closed
• Evolve WatermarkOutputMultiplexer to make it reusable in FLIP-27 Sources: Closed
• Change SourceCoordinator to handle resetToCheckpoint() call after started.: Resolved
• Add the subtask index information to the SourceReaderContext.: Resolved
• Add documentation for new FLIP-27 source interface: Closed
• Improve RecordsWithSplitIds API: Closed
• Port File Sources to FLIP-27 API: Closed
• Allow Split Reader based sources to reuse record batches: Closed
• SourceReaderContext should give access to Configuration and Hostbame: Closed