[FLINK-19934] [FLIP-27 source] add new API: SplitEnumeratorContext.runInCoordinatorThread(Runnable)

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.11.2
• Fix Version/s: 1.11.3, 1.12.0
• Component/s: API / DataStream
• Labels: pull-request-available
• Assignee: Steven Zhen Wu
• Reporter: Steven Zhen Wu
• Created: 03/Nov/20 02:01
• Updated: 12/Dec/20 11:53
• Resolved: 08/Nov/20 15:34

Description

Here is the motivation use case. We are implementing event-time alignment across sources in Iceberg source. Basically, each Iceberg source/enumerator tracks its watermark using min/max timestamps captures in the column stats of the data files.

When the watermark from another source advances, notified source/enumerator can try `assignSplits` as constraints may be satisfied now. This callback is initiated from the coordinator thread from the other source. If we have `SplitEnumeratorContext.execute(Runnable r)` API, we can ensure that all the actions by enumerator and assigner are serialized by the coordinator thread. That can avoid the need of locks.

becket_qin sewen what do you think? cc sundaram

Issue Links

• links to: GitHub Pull Request #13955