[HBASE-15594] [YCSB] Improvements

• Type: Umbrella
• Status: Resolved
• Priority: Critical
• Resolution: Implemented
• Assignee: Unassigned
• Reporter: Michael Stack
• Created: 05/Apr/16 16:43
• Updated: 11/Jun/22 18:27
• Resolved: 11/Jun/22 18:27

Description

Running YCSB and getting good results is an arcane art. For example, in my testing, a few handlers (100) with as many readers as I had CPUs (48), and upping connections on clients to same as #cpus made for 2-3x the throughput. The above config changes came of lore; which configurations need tweaking is not obvious going by their names, there were no indications from the app on where/why we were blocked or on which metrics are important to consider. Nor was any of this stuff written down in docs.

Even still, I am stuck trying to make use of all of the machine. I am unable to overrun a server though 8 client nodes trying to beat up a single node (workloadc, all random-read, with no data returned -p readallfields=false). There is also a strange phenomenon where if I add a few machines, rather than 3x the YCSB throughput when 3 nodes in cluster, each machine instead is doing about 1/3rd.

This umbrella issue is to host items that improve our defaults and noting how to get good numbers running YCSB. In particular, I want to be able to saturate a machine.

Here are the configs I'm currently working with. I've not done the work to figure client-side if they are optimal (weird is how big a difference client-side changes can make – need to fix this). On my 48 cpu machine, I can do about 370k random reads a second from data totally cached in bucketcache. If I short-circuit the user gets so they don't do any work but return immediately, I can do 600k ops a second but the CPUs are at 60-70% only. I cannot get them to go above this. Working on it.

<property>
<name>
hbase.ipc.server.read.threadpool.size
</name>
<value>48</value>
</property>
<property>
<name>
    hbase.regionserver.handler.count
</name>
<value>100</value>
</property>

<property>
<name>
hbase.client.ipc.pool.size
</name>
<value>100</value>
</property>

<property>
<name>
hbase.htable.threads.max
</name>
<value>48</value>
</property>

Issue Links

• relates to: HBASE-14331 a single callQueue related improvements
• relates to: HBASE-14479 Apply the Leader/Followers pattern to RpcServer's Reader
• links to: Review Board (branch-1)

Sub-Tasks

• Port "HADOOP-9956 RPC listener inefficiently assigns connections to readers": Resolved
• Metric for active ipc Readers and make default fraction of cpu count: Resolved
• Regression: Random Read/WorkloadC slower in 1.x than 0.98: Resolved
• Allow selection of RpcSchedulers: Resolved
• Add fast path when doing FIFO that skips queue: Resolved
• Fastpath for the FIFO rpcscheduler: Resolved
• Race condition in new FIFO fastpath from HBASE-16023: Open
• Counters are expensive...: Resolved