Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-11693

Add KafkaSerializationSchema that directly uses ProducerRecord

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Implemented
    • Affects Version/s: None
    • Fix Version/s: 1.9.0
    • Component/s: Connectors / Kafka
    • Environment:

      Kafka 0.11.0.0
      Flink 1.4.0
      flink-connector-kafka-0.11_2.11

    • Release Note:
      Hide
      The modern `FlinkKafkaProducer` supports a new `KafkaSerializationSchema` that will replace `KeyedSerializationSchema` in the long run. This new schema allows directly generating Kafka `ProducerRecords` for sending to Kafka, therefore enabling the user to use all available Kafka features (in the context of Kafka records).
      Show
      The modern `FlinkKafkaProducer` supports a new `KafkaSerializationSchema` that will replace `KeyedSerializationSchema` in the long run. This new schema allows directly generating Kafka `ProducerRecords` for sending to Kafka, therefore enabling the user to use all available Kafka features (in the context of Kafka records).

      Description

      This allows using all fields that Kafka offers on ProducerRecord, for example, timestamps and headers.

      Old Text

      Kafka has introduced notion of Header for messages in version 0.11.0.0 https://issues.apache.org/jira/browse/KAFKA-4208.

      But flink-connector-kafka-0.11_2.11 which supports kafka 0.11.0.0 ignores headers when consuming kafka messages.

      It would be useful in some scenarios, such as distributed log tracing, to support message headers to FlinkKafkaConsumer011 and FlinkKafkaProducer011.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                aljoscha Aljoscha Krettek
                Reporter:
                abareghi Mohammad Abareghi
              • Votes:
                1 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 0.5h
                  0.5h