Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-14460 [Perf Regression] Merge of MVCC and SequenceId (HBASE-8763) slowed Increments, CheckAndPuts, batch operations
  3. HBASE-15091

Forward-port to 1.2+ HBASE-15031 "Fix merge of MVCC and SequenceID performance regression in branch-1.0 for Increments"

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • None
    • 1.2.0, 1.3.0
    • Performance
    • None
    • Reviewed
    • Hide
      UPDATE: This forward port was not necessary. hbase-1.2.0 as it happens does not suffer the performance regression. HBASE-12751 which was added to hbase-1.2.0, actually fixed the performance regression in increment and append too. Ignore the below!!!!

      Increments can be 10x slower (or more) when there is high concurrency since HBase 1.0.0 (HBASE-8763).

      This 'fix' adds back a fast increment but speed is achieved by relaxing row-level consistency for Increments (only). The default remains the old, slow, consistent Increment behavior.

      Set "hbase.increment.fast.but.narrow.consistency" to true in hbase-site.xml to enable 'fast' increments and then rolling restart your cluster. This is a setting the server-side needs to read.

      Intermixing fast increment with other Mutations will give indeterminate results; e.g. a Put and Increment against the same Cell will not always give you the result you expect. Fast Increments are consistent unto themselves. A Get with {@link IsolationLevel#READ_UNCOMMITTED} will return the latest increment value or an Increment of an amount zero will do the same (beware doing Get on a cell that has not been incremented yet -- this will return no results).

      The difference between fastAndNarrowConsistencyIncrement and slowButConsistentIncrement is that the former holds the row lock until the WAL sync completes; this allows us to reason that there are no other writers afoot when we read the current increment value. In this case we do not need to wait on mvcc reads to catch up to writes before we proceed with the read of the current Increment value, the root of the slowdown seen in HBASE-14460. The fast-path also does not wait on mvcc to complete before returning to the client (but the write has been synced and put into memstore before we return).

      Also adds a simple performance test tool that will run against existing cluster. It expects the table to be already created (by default it expects the table 'tableName' with a column family 'columnFamilyName'):

      {code}
      $ ./bin/hbase org.apache.hadoop.hbase.IncrementPerformanceTest
      {code]

      Configure it by passing -D options. Here are the set below:

      2015-12-23 19:33:36,941 INFO [main] hbase.IncrementPerformanceTest: Running test with hbase.zookeeper.quorum=localhost, tableName=tableName, columnFamilyName=columnFamilyName, threadCount=80, incrementCount=10000

      ... so to set the tableName pass -DtableName=SOME_TABLENAME

      Here is an example use of the test tool:

      {code}
      $ time ./bin/hbase --config ~/conf_hbase org.apache.hadoop.hbase.IncrementPerformanceTest -DincrementCount=50000
      {code}

      Show
      UPDATE: This forward port was not necessary. hbase-1.2.0 as it happens does not suffer the performance regression. HBASE-12751 which was added to hbase-1.2.0, actually fixed the performance regression in increment and append too. Ignore the below!!!! Increments can be 10x slower (or more) when there is high concurrency since HBase 1.0.0 ( HBASE-8763 ). This 'fix' adds back a fast increment but speed is achieved by relaxing row-level consistency for Increments (only). The default remains the old, slow, consistent Increment behavior. Set "hbase.increment.fast.but.narrow.consistency" to true in hbase-site.xml to enable 'fast' increments and then rolling restart your cluster. This is a setting the server-side needs to read. Intermixing fast increment with other Mutations will give indeterminate results; e.g. a Put and Increment against the same Cell will not always give you the result you expect. Fast Increments are consistent unto themselves. A Get with {@link IsolationLevel#READ_UNCOMMITTED} will return the latest increment value or an Increment of an amount zero will do the same (beware doing Get on a cell that has not been incremented yet -- this will return no results). The difference between fastAndNarrowConsistencyIncrement and slowButConsistentIncrement is that the former holds the row lock until the WAL sync completes; this allows us to reason that there are no other writers afoot when we read the current increment value. In this case we do not need to wait on mvcc reads to catch up to writes before we proceed with the read of the current Increment value, the root of the slowdown seen in HBASE-14460 . The fast-path also does not wait on mvcc to complete before returning to the client (but the write has been synced and put into memstore before we return). Also adds a simple performance test tool that will run against existing cluster. It expects the table to be already created (by default it expects the table 'tableName' with a column family 'columnFamilyName'): {code} $ ./bin/hbase org.apache.hadoop.hbase.IncrementPerformanceTest {code] Configure it by passing -D options. Here are the set below: 2015-12-23 19:33:36,941 INFO [main] hbase.IncrementPerformanceTest: Running test with hbase.zookeeper.quorum=localhost, tableName=tableName, columnFamilyName=columnFamilyName, threadCount=80, incrementCount=10000 ... so to set the tableName pass -DtableName=SOME_TABLENAME Here is an example use of the test tool: {code} $ time ./bin/hbase --config ~/conf_hbase org.apache.hadoop.hbase.IncrementPerformanceTest -DincrementCount=50000 {code}

    Attachments

      1. 15031.addendum
        0.9 kB
        Michael Stack
      2. 15091v2.branch-1.2.patch
        92 kB
        Michael Stack
      3. 15091v3.branch-1.2.patch
        92 kB
        Michael Stack
      4. 15091v4.branch-1.2.patch
        93 kB
        Michael Stack
      5. 15091v5.branch-1.2.patch
        93 kB
        Michael Stack
      6. 15091v6.branch-1.2.patch
        93 kB
        Michael Stack
      7. 15091v6.branch-1.patch
        93 kB
        Michael Stack
      8. 15091v7.branch-1.2.patch
        109 kB
        Michael Stack
      9. 15091v8.branch-1.2.patch
        116 kB
        Michael Stack
      10. 15091v9.branch-1.2.patch
        117 kB
        Michael Stack
      11. HBASE-15091.png
        49 kB
        Michael Stack
      12. HBASE-15091.v1.branch-1.2.patch
        55 kB
        Heng Chen
      13. HBASE-15091.v9.branch-1.2.patch
        117 kB
        Sean Busbey
      14. HBASE-15091-branch-1.2_v1.patch
        55 kB
        Heng Chen
      15. HBASE-15091-branch-1.2.patch
        55 kB
        Heng Chen

      Activity

        People

          stack Michael Stack
          stack Michael Stack
          Votes:
          0 Vote for this issue
          Watchers:
          5 Start watching this issue

          Dates

            Created:
            Updated:
            Resolved: