Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-6127

Streams should never block infinitely

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.0.0
    • None
    • streams

    Description

      Streams uses three consumer APIs that can block infinite: commitSync(), committed(), and position(). Also KafkaProducer#send() can block. If EOS is enabled, KafkaProducer#initTransactions() also used to block (fixed in KAFKA-6446) and we should double check the code if we handle this case correctly.

      If we block within one operation, the whole StreamThread would block, and the instance does not make any progress, becomes unresponsive (for example, KafkaStreams#close() suffers), and we also might drop out of the consumer group.

      Thanks to [KIP-266|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=75974886,] the Consumer now has non-blocking variants that we can use, but the same is not true of Producer. We can add non-blocking variants to Producer as well, or set the appropriate config options to set the max timeout.

      Of course, we'd also need to be sure the catch the appropriate timeout exceptions.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            mjsax Matthias J. Sax
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment