[SPARK-10644] Applications wait even if free executors are available

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Not A Problem
• Affects Version/s: 1.5.0
• Component/s: Scheduler, Spark Core

Environment

RHEL 6.5 64 bit

• Target Version/s: 1.5.0
• Assignee: Unassigned
• Reporter: Balagopal Nair
• Created: 16/Sep/15 18:47
• Updated: 17/May/20 17:47
• Resolved: 29/Sep/15 03:18

Description

Number of workers: 21
Number of executors: 63

Steps to reproduce:
1. Run 4 jobs each with max cores set to 10
2. The first 3 jobs run with 10 each. (30 executors consumed so far)
3. The 4 th job waits even though there are 33 idle executors.

The reason is that a job will not get executors unless
the total number of EXECUTORS in use < the number of WORKERS

If there are executors available, resources should be allocated to the pending job.