Uploaded image for project: 'Apache Nemo'
  1. Apache Nemo
  2. NEMO-388

Off-heap memory management (reuse ByteBuffer)

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None

      Description

      As part of implementing the off-heap SerializedMemoryStore, off-heap memory management has to be done since JVM does not know about the memory usage outside its heap. Moreover, ByteBuffer reuse should be implemented through the management of memory to reduce the overhead of allocateDirect(). allocateDirect() has significant overhead that it can increase the overall job completion time. There will be three main components to be implemented: 1. MemoryPoolAssigner, 2. MemoryChunk and 3. MemoryPool. Detailed information about the design is attached.

        Attachments

        1. Off-heap Memory Management.pdf
          247 kB
          Haeyoon Cho

          Issue Links

            Activity

              People

              • Assignee:
                Haeyoon Haeyoon Cho
                Reporter:
                Haeyoon Haeyoon Cho
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 20m
                  20m