[SPARK-6076] Fix a potential OOM issue when StorageLevel is MEMORY_AND_DISK_SER

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.2.1
• Fix Version/s: 1.4.0
• Component/s: Block Manager, Spark Core
• Assignee: Shixiong Zhu
• Reporter: Shixiong Zhu
• Created: 28/Feb/15 06:49
• Updated: 17/May/20 18:20
• Resolved: 01/May/15 00:38

Description

When StorageLevel is MEMORY_AND_DISK_SER, it will copy the content from file into memory, then put it into MemoryStore. See: https://github.com/apache/spark/blob/dcd1e42d6b6ac08d2c0736bf61a15f515a1f222b/core/src/main/scala/org/apache/spark/storage/BlockManager.scala#L538

However, if the file is bigger than the free memory, OOM will happen. A better approach is testing if there is enough memory. If not, copyForMemory should not be created, since this is an option operation.

Issue Links

• links to: [Github] Pull Request #4827 (zsxwing)