Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-14350 Procedure V2 Phase 2: Assignment Manager
  3. HBASE-17786

Create LoadBalancer perf-tests (test balancer algorithm decoupled from workload)

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 2.0.0
    • Balancer, proc-v2
    • Hide
      $ bin/hbase org.apache.hadoop.hbase.master.balancer.LoadBalancerPerformanceEvaluation -help
      usage: hbase org.apache.hadoop.hbase.master.balancer.LoadBalancerPerformanceEvaluation <options>
      Options:
       -regions <arg> Number of regions to consider by load balancer. Default: 1000000
       -servers <arg> Number of servers to consider by load balancer. Default: 1000
       -load_balancer <arg> Type of Load Balancer to use. Default:
                              org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer
      Show
      $ bin/hbase org.apache.hadoop.hbase.master.balancer.LoadBalancerPerformanceEvaluation -help usage: hbase org.apache.hadoop.hbase.master.balancer.LoadBalancerPerformanceEvaluation <options> Options:  -regions <arg> Number of regions to consider by load balancer. Default: 1000000  -servers <arg> Number of servers to consider by load balancer. Default: 1000  -load_balancer <arg> Type of Load Balancer to use. Default:                         org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer

    Description

      (Below is a quote from mbertozzi taken from an internal issue that I'm moving out here)

      Add perf tools and keep monitored balancer performance (a BalancerPE-type thing).

      Most of the balancers should be instantiable without requiring a mini-cluster, and it easy to create tons of RegionInfo and ServerNames with a for loop.

      The balancer is just creating a map RegionInfo:ServerName.

      There are two methods to test roundRobinAssignment() and retainAssignment()

      Map<ServerName, List<HRegionInfo>> roundRobinAssignment(
          List<HRegionInfo> regions,
          List<ServerName> servers
        ) throws HBaseIOException; 
      
      Map<ServerName, List<HRegionInfo>> retainAssignment(
          Map<HRegionInfo, ServerName> regions,
          List<ServerName> servers
        ) throws HBaseIOException;
      
      

      There are a bunch of obvious optimization that everyone can see just by looking at the code. (like replacing array with set when we do contains/remove operations). It will be nice to have a baseline and start improving from there.

      Attachments

        1. HBASE-17786.002.patch
          15 kB
          Umesh Agashe
        2. HBASE-17786.002.patch
          15 kB
          Umesh Agashe
        3. HBASE-17786.001.patch
          15 kB
          Umesh Agashe

        Issue Links

          Activity

            People

              uagashe Umesh Agashe
              stack Michael Stack
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: