[HBASE-6466] Enable multi-thread for memstore flush

• Type: Improvement
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.95.2
• Fix Version/s: 0.95.0
• Component/s: regionserver
• Hadoop Flags: Reviewed
• Assignee: Chunhui Shen
• Reporter: Chunhui Shen
• Created: 27/Jul/12 03:13
• Updated: 17/Apr/14 10:15
• Resolved: 09/Feb/13 03:53

Description

If the KV is large or Hlog is closed with high-pressure putting, we found memstore is often above the high water mark and block the putting.

So should we enable multi-thread for Memstore Flush?

Some performance test data for reference,

1.test environment ：
random writting；upper memstore limit 5.6GB;lower memstore limit 4.8GB;400 regions per regionserver；row len=50 bytes, value len=1024 bytes;5 regionserver, 300 ipc handler per regionserver;5 client, 50 thread handler per client for writing

2.test results:

one cacheFlush handler, tps: 7.8k/s per regionserver, Flush:10.1MB/s per regionserver, appears many aboveGlobalMemstoreLimit blocking

two cacheFlush handlers, tps: 10.7k/s per regionserver, Flush:12.46MB/s per regionserver,

200 thread handler per client & two cacheFlush handlers, tps:16.1k/s per regionserver, Flush:18.6MB/s per regionserver

Issue Links

• relates to: HBASE-8752 Backport HBASE-6466 to 0.94