[SPARK-4214] With dynamic allocation, avoid outstanding requests for more executors than pending tasks need

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.2.0
• Fix Version/s: 1.2.0
• Component/s: Spark Core, YARN
• Target Version/s: 1.2.0
• Assignee: Sandy Ryza
• Reporter: Sandy Ryza
• Created: 03/Nov/14 23:34
• Updated: 14/Nov/14 23:58
• Resolved: 14/Nov/14 23:58

Description

Dynamic allocation tries to allocate more executors while we have pending tasks remaining. Our current policy can end up with more outstanding executor requests than needed to fulfill all the pending tasks. Capping the executor requests to the number of cores needed to fulfill all pending tasks would make dynamic allocation behavior less sensitive to settings for maxExecutors.

Issue Links

• links to: [Github] Pull Request #3204 (sryza)