Uploaded image for project: 'Mesos'
  1. Mesos
  2. MESOS-6941

Add support for batch processing of status updates, to increase latency / throughput / cluster scalability.

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: agent, master, scheduler api
    • Labels:
      None

      Description

      Currently, each task has its own independent status update stream. Within an individual stream, updates are sent to schedulers in a serial fashion: the agent will send the N+1th status update only after it receives the scheduler's acknowledgement for the Nth status update.

      This approach slows down throughput substantially and has the potential to backlog status updates when they occur rapidly. Rather, we should add the ability for all available updates on a stream to be sent together (there should probably be a limit to the size of this "batch") so that the scheduler can process them together without incurring the round trip acknowledgement latency in between each update.

      In addition, there may be cases of updates where the scheduler only wants the latest information (e.g. download status per MESOS-2256, or possibly health information). But this should be tackled separately.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                bmahler Benjamin Mahler
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated: