A common complaint with Spark in a multi-tenant environment is that applications have a fixed allocation that doesn't grow and shrink with their resource needs. We're blocked on YARN-1197 for dynamically changing the resources within executors, but we can still allocate and discard whole executors.
It would be useful to have some heuristics that
- Request more executors when many pending tasks are building up
- Discard executors when they are idle
See the latest design doc for more information.