[SPARK-13992] Add support for off-heap caching

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Component/s: Spark Core
• Target Version/s: 2.0.0
• Assignee: Josh Rosen
• Reporter: Josh Rosen
• Created: 18/Mar/16 00:08
• Updated: 02/Apr/16 04:36
• Resolved: 01/Apr/16 21:35

Description

We should add support for caching serialized data off-heap within the same process (i.e. using direct buffers or sun.misc.unsafe).

I'll expand this JIRA later with more detail (filing now as a placeholder).

Issue Links

• is blocked by: SPARK-14135 Add off-heap storage memory bookkeeping support to MemoryManager
• is blocked by: SPARK-13980 Incrementally serialize blocks while unrolling them in MemoryStore
• links to: [Github] Pull Request #11805 (JoshRosen)
• links to: [Github] Pull Request #12126 (lw-lin)