Details
-
Bug
-
Status: Resolved
-
Blocker
-
Resolution: Fixed
-
1.6.0
-
None
Description
When execution evicts storage, we decrement the storage memory in two places:
(1) calls MemoryStore#ensureFreeSpace, which internally calls MemoryManager#releaseStorageMemory for each block it drops. This call lowers the storage memory used by the block size.
A seemingly simple fix is just to remove the line in (2). However, this bug is actually masked by SPARK-12165, so this one must be fixed after that one. Josh actually has an outstanding patch to fix both: https://github.com/apache/spark/pull/10170
Attachments
Issue Links
- blocks
-
SPARK-12165 Execution memory requests may fail to evict storage blocks if storage memory usage is below max memory
- Resolved
- links to