Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-11693

Add KafkaSerializationSchema that directly uses ProducerRecord

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Implemented
    • None
    • 1.9.0
    • Connectors / Kafka
    • Kafka 0.11.0.0
      Flink 1.4.0
      flink-connector-kafka-0.11_2.11

    • Hide
      The modern `FlinkKafkaProducer` supports a new `KafkaSerializationSchema` that will replace `KeyedSerializationSchema` in the long run. This new schema allows directly generating Kafka `ProducerRecords` for sending to Kafka, therefore enabling the user to use all available Kafka features (in the context of Kafka records).
      Show
      The modern `FlinkKafkaProducer` supports a new `KafkaSerializationSchema` that will replace `KeyedSerializationSchema` in the long run. This new schema allows directly generating Kafka `ProducerRecords` for sending to Kafka, therefore enabling the user to use all available Kafka features (in the context of Kafka records).

    Description

      This allows using all fields that Kafka offers on ProducerRecord, for example, timestamps and headers.

      Old Text

      Kafka has introduced notion of Header for messages in version 0.11.0.0 https://issues.apache.org/jira/browse/KAFKA-4208.

      But flink-connector-kafka-0.11_2.11 which supports kafka 0.11.0.0 ignores headers when consuming kafka messages.

      It would be useful in some scenarios, such as distributed log tracing, to support message headers to FlinkKafkaConsumer011 and FlinkKafkaProducer011.

      Attachments

        Issue Links

          Activity

            People

              aljoscha Aljoscha Krettek
              abareghi Mohammad Abareghi
              Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 0.5h
                  0.5h