[FLINK-25696] Introduce metadataConsumer to InitContext in Sink

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.15.0
• Component/s: API / DataStream, Connectors / Kafka, Table Store
• Labels: pull-request-available
• Assignee: Jingsong Lee
• Reporter: Jingsong Lee
• Created: 19/Jan/22 05:49
• Updated: 14/Feb/22 08:29
• Resolved: 09/Feb/22 15:27

Description

In Table Store, we want to get the offsets of kafka writer, only the offset returned by the callback inside the KafkaWriter is accurate, so we need this callback mechanism.

This ticket wants to add metadataConsumer to InitContext in Sink:

/**
 * Returns a metadata consumer, the {@link SinkWriter} can publish metadata events of type
 * {@link MetaT} to the consumer. The consumer can accept metadata events in an asynchronous
 * thread, and the {@link Consumer#accept} method is executed very fast.
 */
default <MetaT> Optional<Consumer<MetaT>> metadataConsumer() {
    return Optional.empty();
}

SinkWriter can get this consumer, and publish metadata to the consumer implemented by table store sink.

Issue Links

• duplicates: FLINK-25184 Introduce setDeliveryCallback to KafkaWriter
• links to: GitHub Pull Request #18412