Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-8354

Add KafkaDeserializationSchema that directly uses ConsumerRecord

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Implemented
    • None
    • 1.8.0
    • Connectors / Kafka
    • Kafka 0.11.0.0
      Flink 1.4.0
      flink-connector-kafka-0.11_2.11

    • For the Flink KafkaConsumers, we introduced a new KafkaDeserializationSchema that gives direct access to the Kafka ConsumerRecord. This subsumes the KeyedSerializationSchema functionality, which is deprecated but still available for now.

    Description

      This allows using all fields that Kafka offers on ConsumerRecord, for example, timestamps and headers.

      Old Text

      Kafka has introduced notion of Header for messages in version 0.11.0.0 https://issues.apache.org/jira/browse/KAFKA-4208.

      But flink-connector-kafka-0.11_2.11 which supports kafka 0.11.0.0 ignores headers when consuming kafka messages.

      It would be useful in some scenarios, such as distributed log tracing, to support message headers to FlinkKafkaConsumer011 and FlinkKafkaProducer011.

      Attachments

        Issue Links

          Activity

            People

              Aegeaner Aegeaner
              abareghi Mohammad Abareghi
              Votes:
              1 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 0.5h
                  0.5h