When execution evicts storage, we decrement the storage memory in two places:
(1) calls MemoryStore#ensureFreeSpace, which internally calls MemoryManager#releaseStorageMemory for each block it drops. This call lowers the storage memory used by the block size.
A seemingly simple fix is just to remove the line in (2). However, this bug is actually masked by
SPARK-12165, so this one must be fixed after that one. Josh actually has an outstanding patch to fix both: https://github.com/apache/spark/pull/10170