Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-18294

Reduce global heap pressure: flush based on heap occupancy

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0
    • Fix Version/s: 2.0.0-beta-2, 2.0.0
    • Component/s: None
    • Labels:
      None
    • Release Note:
      Hide
      A region is flushed if its memory component exceeds the region flush threshold.
      A flush policy decides which stores to flush by comparing the size of the store to a column-family-flush threshold.
      If the overall size of all memstores in the machine exceeds the bounds defined by the administrator (denoted global pressure) a region is selected and flushed.
      HBASE-18294 changes flush decisions to be based on heap-occupancy and not data (key-value) size, consistently across levels. This rolls back some of the changes by HBASE-16747. Specifically,
      (1) RSs, Regions and stores track their overall on-heap and off-heap occupancy,
      (2) A region is flushed when its on-heap+off-heap size exceeds the region flush threshold specified in hbase.hregion.memstore.flush.size,
      (3) The store to be flushed is chosen based on its on-heap+off-heap size
      (4) At the RS level, a flush is triggered when the overall on-heap exceeds the on-heap limit, or when the overall off-heap size exceeds the off-heap limit (low/high water marks).

      Note that when the region flush size is set to XXmb a region flush may be triggered even before writing keys and values of size XX because the total heap occupancy of the region which includes additional metadata exceeded the threshold.
      Show
      A region is flushed if its memory component exceeds the region flush threshold. A flush policy decides which stores to flush by comparing the size of the store to a column-family-flush threshold. If the overall size of all memstores in the machine exceeds the bounds defined by the administrator (denoted global pressure) a region is selected and flushed. HBASE-18294 changes flush decisions to be based on heap-occupancy and not data (key-value) size, consistently across levels. This rolls back some of the changes by HBASE-16747 . Specifically, (1) RSs, Regions and stores track their overall on-heap and off-heap occupancy, (2) A region is flushed when its on-heap+off-heap size exceeds the region flush threshold specified in hbase.hregion.memstore.flush.size, (3) The store to be flushed is chosen based on its on-heap+off-heap size (4) At the RS level, a flush is triggered when the overall on-heap exceeds the on-heap limit, or when the overall off-heap size exceeds the off-heap limit (low/high water marks). Note that when the region flush size is set to XXmb a region flush may be triggered even before writing keys and values of size XX because the total heap occupancy of the region which includes additional metadata exceeded the threshold.

      Description

      A region is flushed if its memory component exceed a threshold (default size is 128MB).
      A flush policy decides whether to flush a store by comparing the size of the store to another threshold (that can be configured with hbase.hregion.percolumnfamilyflush.size.lower.bound).
      Currently the implementation (in both cases) compares the data size (key-value only) to the threshold where it should compare the heap size (which includes index size, and metadata).

        Attachments

        1. HBASE-18294.02.patch
          121 kB
          Eshcar Hillel
        2. HBASE-18294.03.patch
          121 kB
          Eshcar Hillel
        3. HBASE-18294.04.patch
          120 kB
          Eshcar Hillel
        4. HBASE-18294.05.patch
          131 kB
          Eshcar Hillel
        5. HBASE-18294.06.patch
          130 kB
          Eshcar Hillel
        6. HBASE-18294.07.patch
          91 kB
          Eshcar Hillel
        7. HBASE-18294.07.patch
          91 kB
          Eshcar Hillel
        8. HBASE-18294.08.patch
          89 kB
          Eshcar Hillel
        9. HBASE-18294.09.patch
          89 kB
          Eshcar Hillel
        10. HBASE-18294.10.patch
          89 kB
          Eshcar Hillel
        11. HBASE-18294.11.patch
          89 kB
          Eshcar Hillel
        12. HBASE-18294.11.patch
          89 kB
          Eshcar Hillel
        13. HBASE-18294.12.patch
          89 kB
          Eshcar Hillel
        14. HBASE-18294.13.patch
          90 kB
          Eshcar Hillel
        15. HBASE-18294.15.patch
          136 kB
          Eshcar Hillel
        16. HBASE-18294.01.patch
          136 kB
          Eshcar Hillel
        17. HBASE-18294.16.patch
          138 kB
          Eshcar Hillel
        18. HBASE-18294.01.patch
          138 kB
          Eshcar Hillel
        19. HBASE-18294.master.01.patch
          141 kB
          Eshcar Hillel
        20. HBASE-18294.01.patch
          141 kB
          Eshcar Hillel
        21. HBASE-18294.01.patch
          141 kB
          Michael Stack
        22. HBASE-18294.master.01.patch
          133 kB
          Eshcar Hillel
        23. HBASE-18294.01.patch
          133 kB
          Eshcar Hillel
        24. HBASE-18294.01.patch
          132 kB
          Eshcar Hillel
        25. HBASE-18294.01.patch
          133 kB
          Eshcar Hillel
        26. HBASE-18294.01.patch
          140 kB
          Eshcar Hillel
        27. HBASE-18294.01.patch
          141 kB
          Eshcar Hillel
        28. HBASE-18294.branch-2.01.patch
          141 kB
          Eshcar Hillel

          Activity

            People

            • Assignee:
              eshcar Eshcar Hillel
              Reporter:
              eshcar Eshcar Hillel
            • Votes:
              0 Vote for this issue
              Watchers:
              19 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: