[HBASE-15158] Change order in which we do write pipeline operations; do all under row locks!

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Component/s: Performance
• Hadoop Flags: Reviewed
• Release Note: Changed the write pipeline order; made it more rational, easier-to-reason-about doing all updates to WA, MemStore, and mvcc while read/write rowlock is held where before we'd release after WAL append and then do sync and mvcc.
• Assignee: Michael Stack
• Reporter: Michael Stack
• Created: 22/Jan/16 06:25
• Updated: 02/Nov/16 08:04
• Resolved: 08/Feb/16 07:00

Description

Change how we do our write pipeline. I want to do all write pipeline ops under row lock so I lean on this fact fixing performance regression in check-and-set type operations like increment, append, and checkAnd* (see sibling issue HBASE-15082).

To be specific, we write like this now:

# take rowlock
# start mvcc
# append to WAL
# add to memstore
# let go of rowlock
# sync WAL
# in case of error: rollback memstore

Instead, write like this:

# take rowlock
# start mvcc
# append to WAL
# sync WAL
# add to memstore
# let go of rowlock

... no need to do rollback.

The old ordering was put in place because it got better performance in a time when WAL was different and before row locks were read/write (HBASE-12751).

Testing in branch-1 shows that a reordering and skipping mvcc waits gets us back to the performance we had before we unified mvcc and sequenceid (HBASE-8763). Tests in HBASE-15046 show that at the macro level using our usual perf tools, reordering pipeline seems to cause no slowdown (see HBASE-15046). A rough compare of increments with reordered write pipeline seems to have us getting back a bunch of our performance (see tail of https://issues.apache.org/jira/browse/HBASE-15082?focusedCommentId=15111703&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15111703 and subsequent comment).

Issue Links

• is blocked by: HBASE-15186 HBASE-15158 Preamble 1 of 2: fix findbugs, add javadoc, change Region#getReadpoint to #getReadPoint, and some util
• relates to: HBASE-15196 HBASE-15158 Preamble 2 of 2:Add Increment tests
• relates to: HBASE-16971 The passed durability of Append/Increment isn't used in wal sync
• relates to: HBASE-16989 RowProcess#postBatchMutate doesn’t be executed before the mvcc transaction completion
• links to: Link to Review Board