Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-3411

Improve load-balancing of concurrently-submitted drivers across workers

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.1.0
    • Fix Version/s: 1.2.0
    • Component/s: Deploy
    • Labels:
      None
    • Target Version/s:

      Description

      If the waiting driver array is too big, the drivers in it will be dispatched to the first worker we get(if it has enough resources), with or without the Randomization.

      We should do randomization every time we dispatch a driver, in order to better balance drivers.

      Update(2014/9/6):Doing shuffle is much slower, so we use round robin to avoid it.

        Attachments

          Activity

            People

            • Assignee:
              WangTaoTheTonic Tao Wang (Inactive)
              Reporter:
              WangTaoTheTonic Tao Wang (Inactive)
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: