[HBASE-16698] Performance issue: handlers stuck waiting for CountDownLatch inside WALKey#getWriteEntry under high writing workload

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.2.3
• Fix Version/s: 1.4.0, 2.0.0
• Component/s: Performance
• Hadoop Flags: Reviewed

Release Note

Assign sequenceid to an edit before we go on the ringbuffer; undoes contention on WALKey latch. Adds a new config "hbase.hregion.mvcc.preassign" which defaults to true: i.e. this speedup is enabled.

User could set this per-table level, like:
create 'table',{NAME=>'f1',CONFIGURATION=>{'hbase.hregion.mvcc.preassign'=>'false'}}

• Assignee: Yu Li
• Reporter: Yu Li
• Created: 23/Sep/16 19:53
• Updated: 22/Jan/17 17:20
• Resolved: 22/Jan/17 09:25

Description

As titled, on our production environment we observed 98 out of 128 handlers get stuck waiting for the CountDownLatch seqNumAssignedLatch inside WALKey#getWriteEntry under a high writing workload.

After digging into the problem, we found that the problem is mainly caused by advancing mvcc in the append logic. Below is some detailed analysis:

Under current branch-1 code logic, all batch puts will call WALKey#getWriteEntry after appending edit to WAL, and seqNumAssignedLatch is only released when the relative append call is handled by RingBufferEventHandler (see FSWALEntry#stampRegionSequenceId). Because currently we're using a single event handler for the ringbuffer, the append calls are handled one by one (actually lot's of our current logic depending on this sequential dealing logic), and this becomes a bottleneck under high writing workload.

The worst part is that by default we only use one WAL per RS, so appends on all regions are dealt with in sequential, which causes contention among different regions...

To fix this, we could also take use of the "sequential appends" mechanism, that we could grab the WriteEntry before publishing append onto ringbuffer and use it as sequence id, only that we need to add a lock to make "grab WriteEntry" and "append edit" a transaction. This will still cause contention inside a region but could avoid contention between different regions. This solution is already verified in our online environment and proved to be effective.

Notice that for master (2.0) branch since we already change the write pipeline to sync before writing memstore (HBASE-15158), this issue only exists for the ASYNC_WAL writes scenario.

Issue Links

• breaks: HBASE-17471 Region Seqid will be out of order in WAL if using mvccPreAssign
• is related to: HBASE-17509 Remove the mvcc preassign lock and take usage of MVCC internal lock instead
• relates to: HBASE-17482 mvcc mechanism fails when using mvccPreAssign
• relates to: HBASE-16873 WAL: SequenceId assign with less friction
• relates to: HBASE-17507 Add document for the mvcc preassign feature introduced in HBASE-16698