Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-14460

[Perf Regression] Merge of MVCC and SequenceId (HBASE-8763) slowed Increments, CheckAndPuts, batch operations

    XMLWordPrintableJSON

    Details

    • Type: Umbrella
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.2.0, 1.3.0, 1.1.4, 1.0.4, 2.0.0
    • Component/s: Performance
    • Labels:
      None
    • Release Note:
      Hide
      This release note tries to tell the general story. Dive into sub-tasks for more specific release noting.

      Increments, appends, checkAnd* have been slow since hbase-.1.0.0. The unification of mvcc and sequence id done by HBASE-8763 was responsible.

      A ‘fast-path’ workaround was added by HBASE-15031 “Fix merge of MVCC and SequenceID performance regression in branch-1.0 for Increments”. It became available in 1.0.3 and 1.1.3. To enable the fast path, set "hbase.increment.fast.but.narrow.consistency" and then rolling restart. The workaround was for increments only (appends, checkAndPut, etc., were not addressed. See HBASE-15031 release note for more detail).

      Subsequently, the regression was properly identified and fixed in HBASE-15213 and the fix applied to branch-1.0 and branch-1.1. As it happens, hbase-1.2.0 does not suffer from the performance regression (though the thought was that it did -- and so it got the fast-path patch too via HBASE-15092) nor does the master branch. HBASE-15213 identified that HBASE-12751 (as a side effect) had cured the regression.

      hbase-1.0.4 (if it is ever released -- 1.0 has been end-of-lifed) and hbase-1.1.4 will have the HBASE-15213 fix. If you are suffering from the increment regression and you are on 1.0.3 or 1.1.3, you can enable the work around to get back your increment performance but you should upgrade.
      Show
      This release note tries to tell the general story. Dive into sub-tasks for more specific release noting. Increments, appends, checkAnd* have been slow since hbase-.1.0.0. The unification of mvcc and sequence id done by HBASE-8763 was responsible. A ‘fast-path’ workaround was added by HBASE-15031 “Fix merge of MVCC and SequenceID performance regression in branch-1.0 for Increments”. It became available in 1.0.3 and 1.1.3. To enable the fast path, set "hbase.increment.fast.but.narrow.consistency" and then rolling restart. The workaround was for increments only (appends, checkAndPut, etc., were not addressed. See HBASE-15031 release note for more detail). Subsequently, the regression was properly identified and fixed in HBASE-15213 and the fix applied to branch-1.0 and branch-1.1. As it happens, hbase-1.2.0 does not suffer from the performance regression (though the thought was that it did -- and so it got the fast-path patch too via HBASE-15092 ) nor does the master branch. HBASE-15213 identified that HBASE-12751 (as a side effect) had cured the regression. hbase-1.0.4 (if it is ever released -- 1.0 has been end-of-lifed) and hbase-1.1.4 will have the HBASE-15213 fix. If you are suffering from the increment regression and you are on 1.0.3 or 1.1.3, you can enable the work around to get back your increment performance but you should upgrade.

      Description

      As reported by 鈴木俊裕 up on the mailing list – see "Performance degradation between CDH5.3.1(HBase0.98.6) and CDH5.4.5(HBase1.0.0)" – our unification of sequenceid and MVCC slows Increments (and other ops) as the mvcc needs to 'catch up' to our current point before we can read the last Increment value that we need to update.

      We can say that our Increment is just done wrong, we should just be writing Increments and summing on read, but checkAndPut as well as batching operations have the same issue. Fix.

        Attachments

        1. 14460.txt
          4 kB
          Michael Stack
        2. region_lock.png
          230 kB
          Michael Stack
        3. flamegraph-28066.098.singlecell.svg
          565 kB
          Michael Stack
        4. flamegraph-13120.svg.master.singlecell.svg
          911 kB
          Michael Stack
        5. flamegraph-9466.094.singlecell.svg
          463 kB
          Michael Stack
        6. testincrement.master.patch
          5 kB
          Michael Stack
        7. testincrement.098.patch
          5 kB
          Michael Stack
        8. testincrement.094.patch
          5 kB
          Michael Stack
        9. flamegraph-31647.master.100.svg
          641 kB
          Michael Stack
        10. flamegraph-28767.098.100.svg
          821 kB
          Michael Stack
        11. flamegraph-26636.094.100.svg
          494 kB
          Michael Stack
        12. 0.94.test.patch
          9 kB
          Michael Stack
        13. m.test.patch
          7 kB
          Michael Stack
        14. 0.98.test.patch
          7 kB
          Michael Stack
        15. HBASE-14460-discussion.patch
          30 kB
          Jingcheng Du
        16. 1.0.80.flamegraph-7932.svg
          687 kB
          Michael Stack
        17. 98.80.flamegraph-11428.svg
          994 kB
          Michael Stack
        18. client.test.patch
          6 kB
          Michael Stack
        19. hack.flamegraph-16593.svg
          1.01 MB
          Michael Stack
        20. hack.uncommitted.patch
          9 kB
          Michael Stack
        21. 14460.v0.branch-1.0.patch
          55 kB
          Michael Stack

          Issue Links

          There are no Sub-Tasks for this issue.

            Activity

              People

              • Assignee:
                stack Michael Stack
                Reporter:
                stack Michael Stack
              • Votes:
                1 Vote for this issue
                Watchers:
                39 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: