Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-15594

[YCSB] Improvements

    XMLWordPrintableJSON

    Details

    • Type: Umbrella
    • Status: Open
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Running YCSB and getting good results is an arcane art. For example, in my testing, a few handlers (100) with as many readers as I had CPUs (48), and upping connections on clients to same as #cpus made for 2-3x the throughput. The above config changes came of lore; which configurations need tweaking is not obvious going by their names, there were no indications from the app on where/why we were blocked or on which metrics are important to consider. Nor was any of this stuff written down in docs.

      Even still, I am stuck trying to make use of all of the machine. I am unable to overrun a server though 8 client nodes trying to beat up a single node (workloadc, all random-read, with no data returned -p readallfields=false). There is also a strange phenomenon where if I add a few machines, rather than 3x the YCSB throughput when 3 nodes in cluster, each machine instead is doing about 1/3rd.

      This umbrella issue is to host items that improve our defaults and noting how to get good numbers running YCSB. In particular, I want to be able to saturate a machine.

      Here are the configs I'm currently working with. I've not done the work to figure client-side if they are optimal (weird is how big a difference client-side changes can make – need to fix this). On my 48 cpu machine, I can do about 370k random reads a second from data totally cached in bucketcache. If I short-circuit the user gets so they don't do any work but return immediately, I can do 600k ops a second but the CPUs are at 60-70% only. I cannot get them to go above this. Working on it.

      <property>
      <name>
      hbase.ipc.server.read.threadpool.size
      </name>
      <value>48</value>
      </property>
      <property>
      <name>
          hbase.regionserver.handler.count
      </name>
      <value>100</value>
      </property>
      
      <property>
      <name>
      hbase.client.ipc.pool.size
      </name>
      <value>100</value>
      </property>
      
      <property>
      <name>
      hbase.htable.threads.max
      </name>
      <value>48</value>
      </property>
      

        Attachments

        1. counts.patch
          7 kB
          Michael Stack
        2. fast.patch
          5 kB
          Michael Stack
        3. fifo.hits.png
          15 kB
          Michael Stack
        4. fifo.readers.png
          39 kB
          Michael Stack
        5. withcodel.png
          21 kB
          Michael Stack

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                stack Michael Stack
              • Votes:
                0 Vote for this issue
                Watchers:
                19 Start watching this issue

                Dates

                • Created:
                  Updated: