[NEMO-350] Implement Off-heap SerializedMemoryStore

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.2
• Labels: gsoc2019 pull-request-available
• Assignee: Haeyoon Cho
• Reporter: Gyewon Lee
• Created: 07/Mar/19 02:33
• Updated: 07/Mar/20 06:36
• Resolved: 25/Jun/19 06:30

Description

Currently, Nemo SerializedMemoryStore stores its data inside JVM heap, which may cause excessive garbage collection overhead and makes the overall job completion time longer.

 

One way to alleviate this problem is moving data to on-heap region to off-heap region and manages memory allocation by ourselves. To implement this feature, we need to use specialized features such as `ByteBuffer` or `Unsafe`.

Issue Links

• incorporates: NEMO-388 Off-heap memory management (reuse ByteBuffer)
• incorporates: NEMO-383 Implement DirectByteBufferOutputStream for Off-heap SerializedMemoryStore
• incorporates: NEMO-384 Implement DirectByteBufferInputStream for Off-heap SerializedMemoryStore
• links to: GitHub Pull Request #221
• links to: GitHub Pull Request #222