Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-6106

Postpone normal processing of tasks within a thread until restoration of all tasks have completed

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.11.0.1, 1.0.0
    • Fix Version/s: 1.1.1, 2.0.0
    • Component/s: streams
    • Labels:

      Description

      Let's say a stream thread hosts multiple tasks, A and B. At the very beginning when A and B are assigned to the thread, the thread state is TASKS_ASSIGNED, and the thread start restoring these two tasks during this state using the restore consumer while using normal consumer for heartbeating.

      If task A's restoration has completed earlier than task B, then the thread will start processing A immediately even when it is still in the TASKS_ASSIGNED phase. But processing task A will slow down restoration of task B since it is single-thread. So the thread's transition to RUNNING when all of its assigned tasks have completed restoring and now can be processed will be delayed.

      Note that the streams instance's state will only transit to RUNNING when all of its threads have transit to RUNNING, so the instance's transition will also be delayed by this scenario.

      We'd better to not start processing ready tasks immediately, but instead focus on restoration during the TASKS_ASSIGNED state to shorten the overall time of the instance's state transition.

        Attachments

          Activity

            People

            • Assignee:
              ckamal Kamal Chandraprakash
              Reporter:
              guozhang Guozhang Wang
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: