[NEMO-388] Off-heap memory management (reuse ByteBuffer)

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.2
• Labels: GSoC2019 pull-request-available
• Assignee: Haeyoon Cho
• Reporter: Haeyoon Cho
• Created: 22/May/19 10:28
• Updated: 07/Mar/20 06:35
• Resolved: 07/Aug/19 06:31

Description

As part of implementing the off-heap SerializedMemoryStore, off-heap memory management has to be done since JVM does not know about the memory usage outside its heap. Moreover, ByteBuffer reuse should be implemented through the management of memory to reduce the overhead of allocateDirect(). allocateDirect() has significant overhead that it can increase the overall job completion time. There will be three main components to be implemented: 1. MemoryPoolAssigner, 2. MemoryChunk and 3. MemoryPool. Detailed information about the design is attached.

Issue Links

• is part of: NEMO-350 Implement Off-heap SerializedMemoryStore
• relates to: NEMO-413 Fix index checking for byte access of MemoryChunk using UNSAFE
• relates to: NEMO-403 Remove 'transient' uses of SerializedPartition
• relates to: NEMO-412 Address Sonar Cloud issue for MemoryChunk
• relates to: NEMO-397 Separation of JVM heap region and off-heap memory region
• links to: GitHub Pull Request #223