[SPARK-12165] Execution memory requests may fail to evict storage blocks if storage memory usage is below max memory

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 1.6.0
• Component/s: Spark Core
• Target Version/s: 1.6.0
• Assignee: Josh Rosen
• Reporter: Josh Rosen
• Created: 07/Dec/15 01:30
• Updated: 09/Dec/15 19:47
• Resolved: 09/Dec/15 19:40

Description

Consider a scenario where storage memory usage has grown past the size of the unevictable storage region (spark.memory.storageFraction * maxMemory) and a task needs to acquire more execution memory by reclaiming evictable storage memory. If the storage memory usage is less than maxMemory, then there's a possibility that no storage blocks will be evicted. This is caused by how MemoryStore.ensureFreeSpace() is called inside of StorageMemoryPool.shrinkPoolToReclaimSpace().

Here's a failing regression test which demonstrates this bug: https://github.com/apache/spark/commit/b519fe628a9a2b8238dfedbfd9b74bdd2ddc0de4?diff=unified#diff-b3a7cd2e011e048908d70f743c0ed7cfR155

Issue Links

• blocks: SPARK-12155 Execution OOM after a relative large dataset cached in the cluster.
• is blocked by: SPARK-12189 UnifiedMemoryManager double counts storage memory freed
• links to: [Github] Pull Request #10170 (JoshRosen)
• links to: [Github] Pull Request #10229 (andrewor14)