Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-7873

KafkaBasedLog's consumer should always seek to beginning when starting

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.1.0
    • Fix Version/s: 2.2.0, 2.1.1, 2.0.2
    • Component/s: KafkaConnect
    • Labels:
      None

      Description

      KafkaBasedLog expects that callers set the `group.id` for the consumer configuration, and does not itself set the `group.id` if the caller does not explicitly do so. However, KIP-289 changed the default for the `group.id` from a blank string to be null, which changes how KafkaBasedLog behaves when no `group.id` is set, and it actually deprecates and issues a warning when no `group.id` is specified.

      When KafkaBasedLog starts up, it should always start from the beginning of the topic and consume to the end. The consumer's logic for where to start is always:

      1. explicit seek
      2. committed offset (skipped if group.id is null)
      3. auto reset behavior

      and currently Connect does not explicitly seek to the beginning and instead relies upon `auto.offset.reset=earliest`. However, if a `group.id` is specified ant there are committed offsets, then the consumer will start from the committed offsets rather than from the beginning. If a 'group.id' is not specified, then the auto reset behavior should work.

      However, to avoid the warning and possible exception when no `group.id` is specified, KafkaBasedLog should always call consumer.seekToBeginning() during startup.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                rhauch Randall Hauch
                Reporter:
                rhauch Randall Hauch
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: