Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-3188

Deletes in Kafka source should be passed on to KeyedDeserializationSchema

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.0.0
    • 1.0.0
    • Connectors / Kafka
    • None

    Description

      When keys are deleted in the kafka queue, they show up as keys with null payload. Currently in Flink 1.0-SNAPSHOT, these deletions are silently skipped, without increasing current offset.

      This leads to two problems:

      1. When a fetch window contains only deletions, LegacyFetcher gets stuck
      2. For KeyedDeserializationSchemas, it would make sense to pass deletions to the deserializer, so that it can decide to wrap deleted keys as a deletion command. This is also more consistent with the semantics of keys in Kafka queues: When compaction is activated, only the latest message with the same key needs to be kept by Kafka.

      We propose the attached patch as a workaround for both issues.

      Attachments

        1. kafka-deletions.patch
          2 kB
          Sebastian Klemke
        2. testcase.patch
          11 kB
          Sebastian Klemke

        Activity

          People

            rmetzger Robert Metzger
            packet Sebastian Klemke
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: