Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-14918 In-Memory MemStore Flush and Compaction
  3. HBASE-16417

In-Memory MemStore Policy for Flattening and Compactions

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Resolved
    • None
    • 2.0.0
    • None
    • None
    • Hide
      A new memstore compaction policy: Adaptive.
      Adaptive compaction adapts to the workload. It applies either index compaction or data compaction based on the ratio of redundant cells.

      New configuration properties:
      (1) hbase.hregion.compacting.memstore.adaptive.compaction.threshold
      sets the threshold on the fraction of redundant cells to trigger memstore data compaction.
      Default value is 0.5.

      (2) hbase.hregion.compacting.memstore.adaptive.compaction.probability
      sets the initial probability of applying data compaction (a throttling parameter). Default value is 0.5.

      In addition
      - changes the default value of the configuration property
      hbase.memstore.inmemoryflush.threshold.factor to 0.02 (previously 0.25)
       - changes the default value of the configuration property hbase.hregion.compacting.pipeline.segments.limit to 4 (previously 1)
      Show
      A new memstore compaction policy: Adaptive. Adaptive compaction adapts to the workload. It applies either index compaction or data compaction based on the ratio of redundant cells. New configuration properties: (1) hbase.hregion.compacting.memstore.adaptive.compaction.threshold sets the threshold on the fraction of redundant cells to trigger memstore data compaction. Default value is 0.5. (2) hbase.hregion.compacting.memstore.adaptive.compaction.probability sets the initial probability of applying data compaction (a throttling parameter). Default value is 0.5. In addition - changes the default value of the configuration property hbase.memstore.inmemoryflush.threshold.factor to 0.02 (previously 0.25)  - changes the default value of the configuration property hbase.hregion.compacting.pipeline.segments.limit to 4 (previously 1)

    Description

      This Jira explores the performance of different memstore compaction policies.
      It presents the result of write-only workload evaluation as well as read performance in read-write workloads.
      We investigate several settings of hardware (SSD, HDD), key distribution (Zipf, uniform), with multiple settings of the system, and compare measures like write throughput, read latency, write volume, total gc time, etc.

      The submitted patch sets some system properties at the values yielding optimal performance. In addition we suggest a new Adaptive memstore compaction policy that shows good tradeoffs between write throughput and write volume.

      Attachments

        1. HBASE-16417-benchmarkresults-20161101.pdf
          606 kB
          Eshcar Hillel
        2. HBASE-16417-benchmarkresults-20161110.pdf
          870 kB
          Eshcar Hillel
        3. HBASE-16417-benchmarkresults-20161123.pdf
          1.28 MB
          Eshcar Hillel
        4. HBASE-16417-benchmarkresults-20161205.pdf
          1.29 MB
          Eshcar Hillel
        5. HBASE-16417-benchmarkresults-20170309.pdf
          294 kB
          Eshcar Hillel
        6. HBASE-16417-benchmarkresults-20170317.pdf
          446 kB
          Eshcar Hillel
        7. HBASE-16417 - parameter tuning - 20171001.pdf
          240 kB
          Eshcar Hillel
        8. HBASE-16417 - Adaptive Compaction Policy - 20171001.pdf
          796 kB
          Eshcar Hillel
        9. HBASE-16417-V01.patch
          71 kB
          Eshcar Hillel
        10. HBASE-16417.01.patch
          71 kB
          Eshcar Hillel
        11. HBASE-16417.02.patch
          74 kB
          Eshcar Hillel
        12. HBASE-16417.03.patch
          74 kB
          Eshcar Hillel
        13. HBASE-16417.04.patch
          79 kB
          Eshcar Hillel
        14. HBASE-16417.05.patch
          79 kB
          Eshcar Hillel
        15. HBASE-16417.06.patch
          79 kB
          Eshcar Hillel
        16. HBASE-16417.07.patch
          79 kB
          Eshcar Hillel
        17. HBASE-16417.07.patch
          79 kB
          Eshcar Hillel
        18. HBASE-16417.08.patch
          82 kB
          Eshcar Hillel
        19. HBASE-16417.09.patch
          82 kB
          Eshcar Hillel
        20. HBASE-16417.10.patch
          90 kB
          Eshcar Hillel
        21. HBASE-16417.11.patch
          90 kB
          Eshcar Hillel
        22. HBASE-16417.12.patch
          90 kB
          Eshcar Hillel
        23. HBASE-16417.13.patch
          90 kB
          Eshcar Hillel
        24. HBASE-16417.14.patch
          90 kB
          Eshcar Hillel
        25. HBASE-16417.15.patch
          91 kB
          Eshcar Hillel
        26. HBASE-16417.16.patch
          93 kB
          Eshcar Hillel
        27. HBASE-16417.17.patch
          92 kB
          Eshcar Hillel
        28. HBASE-16417.17.patch
          92 kB
          Eshcar Hillel
        29. HBASE-16417.17.patch
          92 kB
          Eshcar Hillel
        30. HBASE-16417.17.patch
          92 kB
          Eshcar Hillel
        31. HBASE-16417.17.patch
          92 kB
          Eshcar Hillel
        32. HBASE-16417.18.patch
          87 kB
          Eshcar Hillel
        33. HBASE-16417.branch-2.17.patch
          92 kB
          Eshcar Hillel

        Issue Links

          Activity

            People

              eshcar Eshcar Hillel
              anastas Anastasia Braginsky
              Votes:
              0 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: