Uploaded image for project: 'Samza'
  1. Samza
  2. SAMZA-2118

Improve the shutdown sequence of AsyncRunLoop.

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.1
    • Component/s: None
    • Labels:
      None

      Description

      A samza container is comprised of multiple tasks and event-loop. Event loop is the main orchestration layer between the different stages of a task(process, commit, window) and delivers messages to task by polling them from different consumers. 

      Currently in samza, the event loop dispatches messages to the tasks as per user-defined configuration `task.max.concurrency`. After dispatching messages with count equal to `task.max.concurrency` to the tasks,  the event loop is blocked until the processing of at-least a single dispatched message finishes(aka the task implementation invokes `taskCallBack.complete()` for at-least a dispatched message). 

      When the event loop is blocked by this, the user thread would sometimes try to shutdown the event loop. This shutdown intent marked by the user-thread will take effect only when the processing of a single message completes. In some-cases, when the task implementation does retry on failure for a message, this could prolong the shutdown of the samza event loop.

      This introduces tight coupling between the task processing time and the event loop shutdown time.

      This behavior was observed predominantly in standalone applications where the samza container is shutdown as a part of the re-balancing phase and task processing time was significantly larger.

      To alleviate this problem, as a part of this ticket, we propose to wake-up the samza event-loop thread when the user-thread marks the intent to shutdown the event-loop.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                spvenkat Shanthoosh Venkataraman
                Reporter:
                spvenkat Shanthoosh Venkataraman
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 50m
                  50m