[FLINK-19934] [FLIP-27 source] add new API: SplitEnumeratorContext.runInCoordinatorThread(Runnable)

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.11.2
• Fix Version/s: 1.12.0, 1.11.3
• Component/s: API / DataStream
• Labels: pull-request-available
• Assignee: Steven Zhen Wu
• Reporter: Steven Zhen Wu
• Created: 03/Nov/20 02:01
• Updated: 12/Dec/20 11:53
• Resolved: 08/Nov/20 15:34

Description

Here is the motivation use case. We are implementing event-time alignment across sources in Iceberg source. Basically, each Iceberg source/enumerator tracks its watermark using min/max timestamps captures in the column stats of the data files.

When the watermark from another source advances, notified source/enumerator can try `assignSplits` as constraints may be satisfied now. This callback is initiated from the coordinator thread from the other source. If we have `SplitEnumeratorContext.execute(Runnable r)` API, we can ensure that all the actions by enumerator and assigner are serialized by the coordinator thread. That can avoid the need of locks.

Jiangjie Qin Stephan Ewen what do you think? cc Sundaram Ananthanarayanan

Issue Links

• links to: GitHub Pull Request #13955