Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-4843

Stream round-robin scheduler is inneficient

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 0.10.2.0
    • 0.11.0.0
    • streams
    • None

    Description

      Currently StreamThread.runloop() uses a simple round-robin scheduler, where a single request is taken from each task for processing, followed by poll, followed by the same process over again. For example, for an app that has just 2 tasks each with 3 records ready to be processed we'd have the following sequence
      poll() -> process 1 request for task T1 -> process 1 request for task T2 -> poll()
      -> process 1 request for task T1 -> process 1 request for task T2 -> poll()
      -> process 1 request for task T1 -> process 1 request for task T2 -> poll()

      This is quite inefficient. Instead, a better round robin scheduler would do:
      poll() -> process all 3 requests for task T1 -> process all 3 requests for task T2 -> poll()

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            enothereska Eno Thereska
            enothereska Eno Thereska
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment