Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-8587 One producer per thread for Kafka Streams EOS
  3. KAFKA-9475

Replace transaction abortion scheduler with a delayed queue

Attach filesAttach ScreenshotVotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Won't Fix
    • None
    • None
    • None
    • None

    Description

      Although we could try setting the txn timeout to be 10 second, the purging scheduler only works every one minute interval, so in the worst case we shall still wait for 1 minute. We are considering several potential fixes:

      1. Change interval to 10 seconds: means we will have 6X frequent checking, more read contention on txn metadata. The benefit here is an easy one-line fix without correctness concern
      2. Use an existing delayed queue, a.k.a purgatory. From what I heard, the purgatory needs at least 2 extra threads to work properly, with some add-on overhead for memory and complexity. The benefit here is more precise timeout reaction, without a redundant full metadata read lock.
      3. Create a new delayed queue. This could be done by using scala delayed queue, the concern here is that whether this approach is production ready. Benefits are the same as 2, with less code complexity potentially

      This ticket is to track #2 progress if we decide to go through this path eventually.  

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            bchen225242 Boyang Chen
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment