[HBASE-6423] Writes should not block reads on blocking updates to memstores

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.94.4, 0.95.0
• Hadoop Flags: Reviewed

Release Note

Added new configuration parameters to adjust the wait time in getting a lock before acting on a region:

hbase.busy.wait.multiplier.max (default 2)
hbase.rowlock.wait.duration (same as the default RPC timeout time)

For reading, it waits at most "hbase.rowlock.wait.duration" in getting a lock.
For writing, it waits at most "hbase.rowlock.wait.duration" * min ( #rows affected, "
hbase.busy.wait.multiplier.max"). However, it waits at most the server side call purge timeout time.

• Assignee: Jimmy Xiang
• Reporter: Karthik Ranganathan
• Created: 18/Jul/12 20:42
• Updated: 02/May/13 02:30
• Resolved: 04/Dec/12 18:17

Description

We have a big data use case where we turn off WAL and have a ton of reads and writes. We found that:

1. flushing a memstore takes a while (GZIP compression)
2. incoming writes cause the new memstore to grow in an unbounded fashion
3. this triggers blocking memstore updates
4. in turn, this causes all the RPC handler threads to block on writes to that memstore
5. we are not able to read during this time as RPC handlers are blocked

At a higher level, we should not hold up the RPC threads while blocking updates, and we should build in some sort of rate control.

Issue Links

• is related to: HBASE-7261 batchMutate/put release the same lock twice
• is related to: HBASE-7376 Acquiring readLock does not apply timeout in HRegion#flushcache
• relates to: HBASE-5162 Basic client pushback mechanism
• relates to: HBASE-5333 Introduce Memstore "backpressure" for writes
• relates to: HBASE-2981 Improve consistency of performance during heavy load