[SPARK-10295] Dynamic allocation in Mesos does not release when RDDs are cached

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.5.0
• Fix Version/s: 1.5.0
• Component/s: Documentation, Spark Core

Environment

Spark 1.5.0 RC1
Centos 6
java 7 oracle

• Assignee: Sean R. Owen
• Reporter: Hans van den Bogert
• Created: 26/Aug/15 15:36
• Updated: 30/Aug/15 01:26
• Resolved: 28/Aug/15 08:32

Description

When running spark in coarse grained mode with shuffle service and dynamic allocation, the driver does not release executors if a dataset is cached.
The console output OTOH shows:
> 15/08/26 17:29:58 WARN SparkContext: Dynamic allocation currently does not support cached RDDs. Cached data for RDD 9 will be lost when executors are removed.
However after the default of 1m, executors are not released. When I perform the same initial setup, loading data, etc, but without caching, the executors are released.

Is this intended behaviour?
If this is intended behaviour, the console warning is misleading.

Issue Links

• links to: [Github] Pull Request #8489 (srowen)