[HBASE-26551] Add FastPath feature to HBase RWQueueRpcExecutor

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.5.0, 1.7.2, 3.0.0-alpha-3
• Component/s: rpc, Scheduler
• Hadoop Flags: Reviewed
• External issue ID: HBASE-17808
• Assignee: Yutong Xiao
• Reporter: Yutong Xiao
• Created: 09/Dec/21 06:53
• Updated: 01/Sep/22 10:57
• Resolved: 18/Jan/22 03:13

Description

In ticket HBASE-17808, the author introduced a fastpath implementation for RWQueueRpcExecutor. It aggregated 3 different independent RpcExecutor to implement the mechanism. This redundancy costed more memory and from its own performance test, it cannot outperform the original implementation. This time, I directly extended RWQueueRpcExecutor to implement the fast path mechanism. From my test result, it has a better queue time performance than before.

YCSB Test:

Constant Configurations:
hbase.regionserver.handler.count: 1000
hbase.ipc.server.callqueue.read.ratio: 0.5
hbase.ipc.server.callqueue.handler.factor: 0.2

Test Workload:

YCSB: 50% Write, 25% Get, 25% Scan. Max Scan length: 1000.
Client Threads: 100

FastPathRWQueueRpcExecutor

RWQueueRpcExecutor

[OVERALL], RunTime(ms), 909365 [OVERALL], Throughput(ops/sec), 5498.3422498116815 [TOTAL_GCS_PS_Scavenge], Count, 1208 [TOTAL_GC_TIME_PS_Scavenge], Time(ms), 8006 [TOTAL_GC_TIME_%_PS_Scavenge], Time(%), 0.8803945610398465 [TOTAL_GCS_PS_MarkSweep], Count, 2 [TOTAL_GC_TIME_PS_MarkSweep], Time(ms), 96 [TOTAL_GC_TIME_%_PS_MarkSweep], Time(%), 0.010556817119638429 [TOTAL_GCs], Count, 1210 [TOTAL_GC_TIME], Time(ms), 8102 [TOTAL_GC_TIME_%], Time(%), 0.8909513781594849 [READ], Operations, 1248885 [READ], AverageLatency(us), 14080.154160711354 [READ], MinLatency(us), 269 [READ], MaxLatency(us), 180735 [READ], 95thPercentileLatency(us), 29775 [READ], 99thPercentileLatency(us), 39391 [READ], Return=OK, 1248885 [CLEANUP], Operations, 200 [CLEANUP], AverageLatency(us), 311.78 [CLEANUP], MinLatency(us), 1 [CLEANUP], MaxLatency(us), 59647 [CLEANUP], 95thPercentileLatency(us), 26 [CLEANUP], 99thPercentileLatency(us), 173 [INSERT], Operations, 1251067 [INSERT], AverageLatency(us), 14235.898240461942 [INSERT], MinLatency(us), 393 [INSERT], MaxLatency(us), 204159 [INSERT], 95thPercentileLatency(us), 29919 [INSERT], 99thPercentileLatency(us), 39647 [INSERT], Return=OK, 1251067 [UPDATE], Operations, 1249582 [UPDATE], AverageLatency(us), 14166.923049467741 [UPDATE], MinLatency(us), 321 [UPDATE], MaxLatency(us), 203647 [UPDATE], 95thPercentileLatency(us), 29855 [UPDATE], 99thPercentileLatency(us), 39551 [UPDATE], Return=OK, 1249582 [SCAN], Operations, 1250466 [SCAN], AverageLatency(us), 30056.68854251135 [SCAN], MinLatency(us), 787 [SCAN], MaxLatency(us), 509183 [SCAN], 95thPercentileLatency(us), 57823 [SCAN], 99thPercentileLatency(us), 74751 [SCAN], Return=OK, 1250466

[OVERALL], RunTime(ms), 958763 [OVERALL], Throughput(ops/sec), 5215.053146606617 [TOTAL_GCS_PS_Scavenge], Count, 1264 [TOTAL_GC_TIME_PS_Scavenge], Time(ms), 8680 [TOTAL_GC_TIME_%_PS_Scavenge], Time(%), 0.9053332262509086 [TOTAL_GCS_PS_MarkSweep], Count, 1 [TOTAL_GC_TIME_PS_MarkSweep], Time(ms), 38 [TOTAL_GC_TIME_%_PS_MarkSweep], Time(%), 0.00396344039142103 [TOTAL_GCs], Count, 1265 [TOTAL_GC_TIME], Time(ms), 8718 [TOTAL_GC_TIME_%], Time(%), 0.9092966666423298 [READ], Operations, 1250961 [READ], AverageLatency(us), 14663.084518222391 [READ], MinLatency(us), 320 [READ], MaxLatency(us), 204415 [READ], 95thPercentileLatency(us), 30815 [READ], 99thPercentileLatency(us), 43071 [READ], Return=OK, 1250961 [CLEANUP], Operations, 200 [CLEANUP], AverageLatency(us), 366.845 [CLEANUP], MinLatency(us), 1 [CLEANUP], MaxLatency(us), 70719 [CLEANUP], 95thPercentileLatency(us), 36 [CLEANUP], 99thPercentileLatency(us), 80 [INSERT], Operations, 1248183 [INSERT], AverageLatency(us), 14334.938754974231 [INSERT], MinLatency(us), 390 [INSERT], MaxLatency(us), 2828287 [INSERT], 95thPercentileLatency(us), 30271 [INSERT], 99thPercentileLatency(us), 41919 [INSERT], Return=OK, 1248183 [UPDATE], Operations, 1250212 [UPDATE], AverageLatency(us), 14283.836318960304 [UPDATE], MinLatency(us), 337 [UPDATE], MaxLatency(us), 2828287 [UPDATE], 95thPercentileLatency(us), 30255 [UPDATE], 99thPercentileLatency(us), 41855 [UPDATE], Return=OK, 1250212 [SCAN], Operations, 1250644 [SCAN], AverageLatency(us), 33153.01709839091 [SCAN], MinLatency(us), 742 [SCAN], MaxLatency(us), 645119 [SCAN], 95thPercentileLatency(us), 62879 [SCAN], 99thPercentileLatency(us), 80447 [SCAN], Return=OK, 1250644

In the metrics screenshot, the first interval is the performance of RWQueueRpcExecutor and the second interval is FastPathRWQueueRpcExecutor.

Issue Links

• is a clone of: HBASE-17808 FastPath for RWQueueRpcExecutor
• links to: GitHub Pull Request #3929
• links to: GitHub Pull Request #4027
• links to: GitHub Pull Request #4028
• links to: GitHub Pull Request #4036