Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-11693

Add KafkaSerializationSchema that directly uses ProducerRecord

Agile BoardRank to TopRank to BottomAttach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskConvert to sub-taskLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Implemented
    • Affects Version/s: None
    • Fix Version/s: 1.9.0
    • Component/s: Connectors / Kafka
    • Environment:

      Kafka 0.11.0.0
      Flink 1.4.0
      flink-connector-kafka-0.11_2.11

    • Release Note:
      Hide
      The modern `FlinkKafkaProducer` supports a new `KafkaSerializationSchema` that will replace `KeyedSerializationSchema` in the long run. This new schema allows directly generating Kafka `ProducerRecords` for sending to Kafka, therefore enabling the user to use all available Kafka features (in the context of Kafka records).
      Show
      The modern `FlinkKafkaProducer` supports a new `KafkaSerializationSchema` that will replace `KeyedSerializationSchema` in the long run. This new schema allows directly generating Kafka `ProducerRecords` for sending to Kafka, therefore enabling the user to use all available Kafka features (in the context of Kafka records).

      Description

      This allows using all fields that Kafka offers on ProducerRecord, for example, timestamps and headers.

      Old Text

      Kafka has introduced notion of Header for messages in version 0.11.0.0 https://issues.apache.org/jira/browse/KAFKA-4208.

      But flink-connector-kafka-0.11_2.11 which supports kafka 0.11.0.0 ignores headers when consuming kafka messages.

      It would be useful in some scenarios, such as distributed log tracing, to support message headers to FlinkKafkaConsumer011 and FlinkKafkaProducer011.

        Attachments

        Issue Links

          Activity

          $i18n.getText('security.level.explanation', $currentSelection) Viewable by All Users
          Cancel

            People

            • Assignee:
              aljoscha Aljoscha Krettek
              Reporter:
              abareghi Mohammad Abareghi

              Dates

              • Created:
                Updated:
                Resolved:

              Time Tracking

              Estimated:
              Original Estimate - Not Specified
              Not Specified
              Remaining:
              Remaining Estimate - 0h
              0h
              Logged:
              Time Spent - 0.5h
              0.5h

                Issue deployment