Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-6106

Postpone normal processing of tasks within a thread until restoration of all tasks have completed



    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s:, 1.0.0
    • Fix Version/s: 1.1.1, 2.0.0
    • Component/s: streams
    • Labels:


      Let's say a stream thread hosts multiple tasks, A and B. At the very beginning when A and B are assigned to the thread, the thread state is TASKS_ASSIGNED, and the thread start restoring these two tasks during this state using the restore consumer while using normal consumer for heartbeating.

      If task A's restoration has completed earlier than task B, then the thread will start processing A immediately even when it is still in the TASKS_ASSIGNED phase. But processing task A will slow down restoration of task B since it is single-thread. So the thread's transition to RUNNING when all of its assigned tasks have completed restoring and now can be processed will be delayed.

      Note that the streams instance's state will only transit to RUNNING when all of its threads have transit to RUNNING, so the instance's transition will also be delayed by this scenario.

      We'd better to not start processing ready tasks immediately, but instead focus on restoration during the TASKS_ASSIGNED state to shorten the overall time of the instance's state transition.




            • Assignee:
              ckamal Kamal Chandraprakash
              guozhang Guozhang Wang
            • Votes:
              0 Vote for this issue
              6 Start watching this issue


              • Created: