Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-6906

Kafka Streams does not commit transactions if data is produced via wall-clock punctuation

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.1.0
    • 0.11.0.3, 1.0.2, 1.1.1, 2.0.0
    • streams
    • None

    Description

      Committing in Kafka Streams happens in regular intervals. However, committing only happens if new input records got processed since the last commit (via setting flag `commitOffsetNeeded` within `StreamTask#process()`)

      However, data could also be emitted via wall-clock based punctuation calls. Especially if EOS is enabled, this is an issue (maybe also for non-EOS) because the current running transaction is not committed and thus might time out leading to a fatal error.

      Attachments

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. KAFKA-6838 Transaction timeout after punctuation results in ProducerFencedException

          • Major - Major loss of function.
          • Resolved
          relates to

          Bug - A problem which impairs or prevents the functions of the product. KAFKA-5510 Streams should commit all offsets regularly

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link GitHub Pull Request #5105

          Web Link GitHub Pull Request #5196

          Activity

            People

              jadireddi Jagadesh Adireddi
              mjsax Matthias J. Sax
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: