[SPARK-3000] Drop old blocks to disk in parallel when memory is not large enough for caching new blocks

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Auto Closed
• Affects Version/s: 1.1.0
• Component/s: Block Manager, Spark Core
• Labels: bulk-closed
• Assignee: Josh Rosen
• Reporter: Zhang, Liye
• Created: 13/Aug/14 05:58
• Updated: 06/Jun/19 13:58
• Resolved: 06/Jun/19 13:58

Description

In spark, rdd can be cached in memory for later use, and the cached memory size is "spark.executor.memory * spark.storage.memoryFraction" for spark version before 1.1.0, and "spark.executor.memory * spark.storage.memoryFraction * spark.storage.safetyFraction" after SPARK-1777.

For Storage level MEMORY_AND_DISK, when free memory is not enough to cache new blocks, old blocks might be dropped to disk to free up memory for new blocks. This operation is processed by ensureFreeSpace in MemoryStore.scala, there will always be a "accountingLock" held by the caller to ensure only one thread is dropping blocks. This method can not fully used the disks throughput when there are multiple disks on the working node. When testing our workload, we found this is really a bottleneck when size of old blocks to be dropped is really large.

We have tested the parallel method on spark 1.0, the speedup is significant. So it's necessary to make dropping blocks operation in parallel.

Issue Links

• is duplicated by: SPARK-1888 enhance MEMORY_AND_DISK mode by dropping blocks in parallel
• links to: [Github] Pull Request #2134 (liyezhang556520)
• links to: [Github] Pull Request #11874 (JoshRosen)