[SPARK-4352] Incorporate locality preferences in dynamic allocation requests

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.2.0
• Fix Version/s: 1.5.0
• Component/s: Spark Core, YARN
• Target Version/s: 1.5.0
• Assignee: Saisai Shao
• Reporter: Sandy Ryza
• Created: 12/Nov/14 01:05
• Updated: 08/Aug/16 05:05
• Resolved: 27/Jul/15 22:49

Description

Currently, achieving data locality in Spark is difficult unless an application takes resources on every node in the cluster. preferredNodeLocalityData provides a sort of hacky workaround that has been broken since 1.0.

With dynamic executor allocation, Spark requests executors in response to demand from the application. When this occurs, it would be useful to look at the pending tasks and communicate their location preferences to the cluster resource manager.

Issue Links

• is related to: SPARK-1714 Take advantage of AMRMClient APIs to simplify logic in YarnAllocationHandler
• is related to: SPARK-9817 Improve the container placement strategy by considering the localities of pending container requests
• relates to: SPARK-16944 [MESOS] Improve data locality when launching new executors when dynamic allocation is enabled
• links to: [Github] Pull Request #6394 (jerryshao)