HBase
  1. HBase
  2. HBASE-10191 Move large arena storage off heap
  3. HBASE-10655

ByteRange-fronted slab allocator for on- and off-heap cellblock storage

    Details

    • Type: Sub-task Sub-task
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Our ByteRange interface has some nice properties such as a compareTo method using hbase-friendly unsigned byte comparison. This can be backed by either on-heap or off-heap byte buffers. Consider an allocator that internally allocates huge on or off-heap slabs and hands out ByteRanges over portions of those arenas. Keep it as simple as possible, optimized for HBase needs. Can be fairly standalone with respect to the rest of the HBase code.

        Issue Links

          Activity

          Hide
          Andrew Purtell added a comment -

          On HBASE-10191, Matt Corgan suggested the allocator could also do constant small (sequential) compactions.

          We can maintain new metrics on numbers of buffers allocated, stats on arenas, stats on wastage and internal fragmentation of the buffers, etc, and use these to guide additional optimizations and refinements.

          Show
          Andrew Purtell added a comment - On HBASE-10191 , Matt Corgan suggested the allocator could also do constant small (sequential) compactions. We can maintain new metrics on numbers of buffers allocated, stats on arenas, stats on wastage and internal fragmentation of the buffers, etc, and use these to guide additional optimizations and refinements.
          Hide
          Andrew Purtell added a comment -

          HDFS-6709 introduces a slab allocator as hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/SlabAllocator.java . It's simple but can be easily adapted for this issue as a starting point for further refinements. The hadoop-common code introduces a SlabRegion type that is very similar to our PositionedByteRange.

          Show
          Andrew Purtell added a comment - HDFS-6709 introduces a slab allocator as hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/SlabAllocator.java . It's simple but can be easily adapted for this issue as a starting point for further refinements. The hadoop-common code introduces a SlabRegion type that is very similar to our PositionedByteRange.

            People

            • Assignee:
              Unassigned
              Reporter:
              Andrew Purtell
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:

                Development