-
Type:
Bug
-
Status: Resolved
-
Priority:
Blocker
-
Resolution: Fixed
-
Affects Version/s: 1.6.0
-
Fix Version/s: 1.6.0
-
Component/s: Spark Core
-
Labels:None
-
Target Version/s:
When execution evicts storage, we decrement the storage memory in two places:
(1) calls MemoryStore#ensureFreeSpace, which internally calls MemoryManager#releaseStorageMemory for each block it drops. This call lowers the storage memory used by the block size.
A seemingly simple fix is just to remove the line in (2). However, this bug is actually masked by SPARK-12165, so this one must be fixed after that one. Josh actually has an outstanding patch to fix both: https://github.com/apache/spark/pull/10170
- blocks
-
SPARK-12165 Execution memory requests may fail to evict storage blocks if storage memory usage is below max memory
-
- Resolved
-
- links to