Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-4277

Support external shuffle service on Worker

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.2.0
    • Fix Version/s: 1.2.0
    • Component/s: Deploy, Spark Core
    • Labels:
      None
    • Target Version/s:

      Description

      It's less useful to have an external shuffle service on the Spark Standalone Worker than on YARN or Mesos (as executor allocations tend to be more static), but it would be good to help test the code path. It would also make Spark more resilient to particular executor failures.

      Cool side-feature: When SPARK-4236 is fixed and integrated, the Worker will take care of cleaning up executor directories, which will mean executors terminate more quickly and that we don't leak data if the executor dies forcefully.

        Attachments

          Activity

            People

            • Assignee:
              ilikerps Aaron Davidson
              Reporter:
              ilikerps Aaron Davidson
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: