Details
-
Improvement
-
Status: Resolved
-
Critical
-
Resolution: Fixed
-
1.2.0
-
None
Description
Currently, achieving data locality in Spark is difficult unless an application takes resources on every node in the cluster. preferredNodeLocalityData provides a sort of hacky workaround that has been broken since 1.0.
With dynamic executor allocation, Spark requests executors in response to demand from the application. When this occurs, it would be useful to look at the pending tasks and communicate their location preferences to the cluster resource manager.
Attachments
Attachments
Issue Links
- is related to
-
SPARK-1714 Take advantage of AMRMClient APIs to simplify logic in YarnAllocationHandler
- Resolved
-
SPARK-9817 Improve the container placement strategy by considering the localities of pending container requests
- Resolved
- relates to
-
SPARK-16944 [MESOS] Improve data locality when launching new executors when dynamic allocation is enabled
- Resolved
- links to