Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-8650

Streams does not work as expected with auto.offset.reset=none

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.3.0
    • Fix Version/s: None
    • Component/s: streams
    • Labels:

      Description

      The auto.offset.reset policy of none is useful as a safety measure, especially when

      • exactly-once processing is desired, or
      • at-least-once is desired, but it is expensive to reprocess from the beginning.

      In this case, using "none" forces the ops team to explicitly set the offset before the stream can re-start processing, in the (hopefully rare) situations in which the stream consumer offset has been lost for some reason, or in the case of a new stream that should not start processing from the beginning or the end, but somewhere in the middle (this scenario might occur during topic migrations).

      Kafka streams really only supports auto.offset.reset of earliest or latest (see the `Topology.AutoOffsetReset` enum). It is also possible to use the auto.offset.reset configuration value, but this works suboptimally because if the streams application reset tool is used (even with a specific offset specified), the offset is set for the input topic, but it is not, and cannot be, set for the internal topics, which won't exist yet.

      The internal topics are created by Kafka streams at startup time, but because the auto.offset.reset policy of "none" is passed to the consumer of those internal topics, the Kafka stream fails to start with a "NoOffsetForPartitionException".

      Proposals / options:

      1) Allow auto.offset.reset=none to be specified in Consumed.with() so that it affects the input topics, but not the internal topics.

      2) Allow streams to be configured with auto.offset.reset=none, but explicitly set the offset to 0 for newly created internal topics.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              rocketraman Raman Gupta
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated: