Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-14719

Making Semantic configurable in Flinkkafkaproducer to support exactly-once semantic in Table API

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Not A Problem
    • Affects Version/s: 1.8.0
    • Fix Version/s: None
    • Component/s: Connectors / Kafka
    • Labels:
      None

      Description

      Flink supports kafka transaction with FlinkKafkaProducer and FlinkKafkaProducer011 . When we use Datastream API , it's able to realize exactly once semantic . But when we use Table API, things are different.

      The createKafkaProducer method in KafkaTableSink is used to create FlinkKafkaProducer to sending messages to Kafka server. It's like :

      protected SinkFunction<Row> createKafkaProducer(
      		String topic,
      		Properties properties,
      		SerializationSchema<Row> serializationSchema,
      		Optional<FlinkKafkaPartitioner<Row>> partitioner) {
      		return new FlinkKafkaProducer<>(
      			topic,
      			new KeyedSerializationSchemaWrapper<>(serializationSchema),
      			properties,
      			partitioner);
      	}
      

      when we get into the constructor of FlinkKafkaProducer we can see this will lead to an at_least_once semantic producer :

      	public FlinkKafkaProducer(
      		String defaultTopicId,
      		KeyedSerializationSchema<IN> serializationSchema,
      		Properties producerConfig,
      		Optional<FlinkKafkaPartitioner<IN>> customPartitioner) {
      		this(
      			defaultTopicId,
      			serializationSchema,
      			producerConfig,
      			customPartitioner,
      			FlinkKafkaProducer.Semantic.AT_LEAST_ONCE,
      			DEFAULT_KAFKA_PRODUCERS_POOL_SIZE);
      	}
      

      This makes user could not achieve exactly-once semantic when using Table API.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              chaiyq chaiyongqiang
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: