Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-14918 In-Memory MemStore Flush and Compaction
  3. HBASE-14921

Inmemory Compaction Optimizations; Segment Structure

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 2.0.0
    • 2.0.0
    • None
    • None
    • Reviewed
    • Hide
      A long, working issue that discussed Segment formats introducing CellArrayMap (delivered as the patch attached to this issue) and CellChunkMap (to be delivered later in HBASE-16421 but see patch v02 for an embryonic form named CellBlockSerialized); when to copy Segment data (and when not too); and then what to include at flush time (the suffix Segment or all Segments). Designs that evolved as discussion went on are attached. Outstanding issues turned up here, not including a CellChunkMap implementation, are listed below but are to be addressed in follow-ons (See HBASE-16417):

      1. The flattening without compaction is causing many small segments in pipeline, and they are not flushed all together.
      2. The issue of compaction prediction cost.
      Show
      A long, working issue that discussed Segment formats introducing CellArrayMap (delivered as the patch attached to this issue) and CellChunkMap (to be delivered later in HBASE-16421 but see patch v02 for an embryonic form named CellBlockSerialized); when to copy Segment data (and when not too); and then what to include at flush time (the suffix Segment or all Segments). Designs that evolved as discussion went on are attached. Outstanding issues turned up here, not including a CellChunkMap implementation, are listed below but are to be addressed in follow-ons (See HBASE-16417 ): 1. The flattening without compaction is causing many small segments in pipeline, and they are not flushed all together. 2. The issue of compaction prediction cost.

    Description

      Memory optimizations including compressed format representation and offheap allocations

      Attachments

        1. CellBlocksSegmentInMemStore.pdf
          249 kB
          Anastasia Braginsky
        2. CellBlocksSegmentinthecontextofMemStore(1).pdf
          288 kB
          Anastasia Braginsky
        3. HBASE-14921-V01.patch
          36 kB
          Anastasia Braginsky
        4. HBASE-14921-V02.patch
          36 kB
          Anastasia Braginsky
        5. HBASE-14921-V03.patch
          106 kB
          Anastasia Braginsky
        6. HBASE-14921-V04-CA.patch
          96 kB
          Anastasia Braginsky
        7. HBASE-14921-V04-CA-V02.patch
          110 kB
          Anastasia Braginsky
        8. HBASE-14921-V05-CAO.patch
          110 kB
          Anastasia Braginsky
        9. HBASE-14921-V06-CAO.patch
          111 kB
          Anastasia Braginsky
        10. HBASE-14921-V08-CAO.patch
          138 kB
          Anastasia Braginsky
        11. HBASE-14921-V09-CAO.patch
          138 kB
          Anastasia Braginsky
        12. HBASE-14921-V10-CAO.patch
          140 kB
          Anastasia Braginsky
        13. HBASE-14921-V11-CAO.patch
          139 kB
          Anastasia Braginsky
        14. HBASE-14921-V11-CAO.patch
          139 kB
          Anastasia Braginsky
        15. HBASE-14921-V12-CAO.patch
          139 kB
          Anastasia Braginsky
        16. InitialCellArrayMapEvaluation.pdf
          132 kB
          Anastasia Braginsky
        17. IntroductiontoNewFlatandCompactMemStore.pdf
          185 kB
          Anastasia Braginsky
        18. MemstoreItrCountissue.patch
          3 kB
          ramkrishna.s.vasudevan
        19. MemStoreSizes.pdf
          138 kB
          Anastasia Braginsky
        20. NewCompactingMemStoreFlow.pptx
          80 kB
          Anastasia Braginsky

        Issue Links

          relates to

          Sub-task - The sub-task of the issue HBASE-15513 hbase.hregion.memstore.chunkpool.maxsize is 0.0 by default

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HBASE-16645 Wrong range of Cells is caused by CellFlatMap#tailMap, headMap, and SubMap

          • Major - Major loss of function.
          • Closed
          links to

          Web Link Review Board

          Activity

            People

              anastas Anastasia Braginsky
              eshcar Eshcar Hillel
              Votes:
              0 Vote for this issue
              Watchers:
              19 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: