Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-19934

[FLIP-27 source] add new API: SplitEnumeratorContext.runInCoordinatorThread(Runnable)

    XMLWordPrintableJSON

    Details

      Description

      Here is the motivation use case. We are implementing event-time alignment across sources in Iceberg source. Basically, each Iceberg source/enumerator tracks its watermark using min/max timestamps captures in the column stats of the data files.

      When the watermark from another source advances, notified source/enumerator can try `assignSplits` as constraints may be satisfied now. This callback is initiated from the coordinator thread from the other source. If we have `SplitEnumeratorContext.execute(Runnable r)` API, we can ensure that all the actions by enumerator and assigner are serialized by the coordinator thread. That can avoid the need of locks.

      Jiangjie Qin Stephan Ewen what do you think? cc Sundaram Ananthanarayanan

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                stevenz3wu Steven Zhen Wu
                Reporter:
                stevenz3wu Steven Zhen Wu
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: