Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-17455

`TaskCorruptedException` After Client Quota Throttling

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.8.0
    • None
    • clients, streams
    • None

    Description

      When running a Kafka Streams EOS app that goes slightly above a configured user quota, we can reliably reproduce `TaskCorruptedException`s after throttling. This is the case even with an application that goes only 5-10% above the configured quota.

       

      The root cause is a `TimeoutException` encountered in the `TaskExecutor.commitOffsetsOrTransaction`.

       

      Stacktrace provided below:

       

      ```

      19:45:28 ERROR [KAFKA] TaskExecutor - stream-thread [basic-tls-0-core-StreamThread-2] Committing task(s) 1_2 failed. org.apache.kafka.common.errors.TimeoutException: Timeout expired after 60000ms while awaiting AddOffsetsToTxn 19:45:28 WARN [KAFKA] StreamThread - stream-thread [basic-tls-0-core-StreamThread-2] Detected the states of tasks [1_2] are corrupted. Will close the task as dirty and re-create and bootstrap from scratch. org.apache.kafka.streams.errors.TaskCorruptedException: Tasks [1_2] are corrupted and hence need to be re-initialized at org.apache.kafka.streams.processor.internals.TaskExecutor.commitOffsetsOrTransaction(TaskExecutor.java:249) ~[server.jar:?] at org.apache.kafka.streams.processor.internals.TaskExecutor.commitTasksAndMaybeUpdateCommittableOffsets(TaskExecutor.java:154) ~[server.jar:?] at org.apache.kafka.streams.processor.internals.TaskManager.commitTasksAndMaybeUpdateCommittableOffsets(TaskManager.java:1915) ~[server.jar:?] at org.apache.kafka.streams.processor.internals.TaskManager.commit(TaskManager.java:1882) ~[server.jar:?] at org.apache.kafka.streams.processor.internals.StreamThread.maybeCommit(StreamThread.java:1384) ~[server.jar:?] at org.apache.kafka.streams.processor.internals.StreamThread.runOnceWithoutProcessingThreads(StreamThread.java:1033) ~[server.jar:?] at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:711) [server.jar:?] at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:670) [server.jar:?]

      ```

      Attachments

        Activity

          People

            Unassigned Unassigned
            coltmcnealy-lh Colt McNealy
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated: