Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-12499

Adjust transaction timeout according to commit interval on Streams EOS

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • None
    • 3.0.0
    • streams
    • None

    Description

      The transaction timeout is set to 1 minute by default on Producer today, while the commit interval on the other hand could be set to a very large value, which makes the stream always hit transaction timeout and drop into rebalance. We should increase the transaction timeout correspondingly when commit interval is large.

      On the other hand, broker could have a limit on the max transaction timeout to be set. If we scale up client transaction timeout over the limit, stream will fail due to  INVALID_TRANSACTION_TIMEOUT. To alleviate this problem, user could define their own customized transaction timeout to avoid hitting the limit, so we should still respect what user configures in the override.

      The new rule for configuring transaction timeout should look like:
      1. If transaction timeout is set in streams config, use it

      2. if not, transaction_timeout = max(default_transaction_timeout, 10 * commit_interval) 

      Additionally if INVALID_TRANSACTION_TIMEOUT was thrown on Streams when calling initTransaction(), we should wrap the exception to inform user that their setting for commit interval could potentially be too high and should adjust.

      Attachments

        Issue Links

          Activity

            People

              bchen225242 Boyang Chen
              bchen225242 Boyang Chen
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: