Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-8354

Add KafkaDeserializationSchema that directly uses ConsumerRecord

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Implemented
    • Affects Version/s: None
    • Fix Version/s: 1.8.0
    • Component/s: Connectors / Kafka
    • Environment:

      Kafka 0.11.0.0
      Flink 1.4.0
      flink-connector-kafka-0.11_2.11

    • Release Note:
      For the Flink KafkaConsumers, we introduced a new KafkaDeserializationSchema that gives direct access to the Kafka ConsumerRecord. This subsumes the KeyedSerializationSchema functionality, which is deprecated but still available for now.

      Description

      This allows using all fields that Kafka offers on ConsumerRecord, for example, timestamps and headers.

      Old Text

      Kafka has introduced notion of Header for messages in version 0.11.0.0 https://issues.apache.org/jira/browse/KAFKA-4208.

      But flink-connector-kafka-0.11_2.11 which supports kafka 0.11.0.0 ignores headers when consuming kafka messages.

      It would be useful in some scenarios, such as distributed log tracing, to support message headers to FlinkKafkaConsumer011 and FlinkKafkaProducer011.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Aegeaner Aegeaner
                Reporter:
                abareghi Mohammad Abareghi
              • Votes:
                1 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 0.5h
                  0.5h