Error Message expected:<3.0> but was:<7.0> Stacktrace java.lang.AssertionError: expected:<3.0> but was:<7.0> at org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancer.testMoveCostMultiplier(TestStochasticLoadBalancer.java:223) Standard Output 2020-07-20 23:53:17,509 INFO [Time-limited test] hbase.ResourceChecker(151): before: master.balancer.TestStochasticLoadBalancer#testNeedBalance Thread=10, OpenFileDescriptor=216, MaxFileDescriptor=1048576, SystemLoadAverage=637, ProcessCount=14, AvailableMemoryMB=12094 2020-07-20 23:53:17,512 INFO [Time-limited test] balancer.BaseLoadBalancer(1053): slop=0.0, systemTablesOnMaster=false 2020-07-20 23:53:17,512 INFO [Time-limited test] balancer.StochasticLoadBalancer(224): Loaded config; maxSteps=1000000, stepsPerRegion=800, maxRunningTime=30000, isByTable=true, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] etc. 2020-07-20 23:53:17,543 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table0 2020-07-20 23:53:17,545 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table5 2020-07-20 23:53:17,545 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table4 2020-07-20 23:53:17,545 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table3 2020-07-20 23:53:17,546 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table2 2020-07-20 23:53:17,546 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table1 2020-07-20 23:53:17,546 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table0 2020-07-20 23:53:17,546 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table9 2020-07-20 23:53:17,548 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table8 2020-07-20 23:53:17,548 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table7 2020-07-20 23:53:17,549 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table6 2020-07-20 23:53:17,550 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table1 2020-07-20 23:53:17,558 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,559 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table1); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,559 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table0 2020-07-20 23:53:17,560 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,560 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table0); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,561 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table1 2020-07-20 23:53:17,562 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,562 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table1); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,563 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table0 2020-07-20 23:53:17,564 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 2 regions. 2020-07-20 23:53:17,564 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table0); total cost=15.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,565 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table1 2020-07-20 23:53:17,566 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 2 regions. 2020-07-20 23:53:17,566 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table1); total cost=15.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,567 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table0 2020-07-20 23:53:17,567 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 2 regions. 2020-07-20 23:53:17,568 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table0); total cost=15.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,569 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table2 2020-07-20 23:53:17,569 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,570 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table2); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,570 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table1 2020-07-20 23:53:17,571 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 2 regions. 2020-07-20 23:53:17,571 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table1); total cost=15.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,572 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table0 2020-07-20 23:53:17,572 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 2 regions. 2020-07-20 23:53:17,573 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table0); total cost=15.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,574 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table3 2020-07-20 23:53:17,574 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,575 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table3); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,575 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table2 2020-07-20 23:53:17,576 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,577 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table2); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,577 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table1 2020-07-20 23:53:17,578 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 2 regions. 2020-07-20 23:53:17,578 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table1); total cost=15.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,578 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table0 2020-07-20 23:53:17,579 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 2 regions. 2020-07-20 23:53:17,580 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table0); total cost=15.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,580 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table0 2020-07-20 23:53:17,581 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 2 regions. 2020-07-20 23:53:17,581 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table0); total cost=15.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,582 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table0 2020-07-20 23:53:17,582 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,583 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table0); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,584 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table5 2020-07-20 23:53:17,585 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,585 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table5); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,585 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table4 2020-07-20 23:53:17,586 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,587 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table4); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,587 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table3 2020-07-20 23:53:17,587 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,588 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table3); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,588 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table2 2020-07-20 23:53:17,588 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,589 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table2); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,589 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table1 2020-07-20 23:53:17,589 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,590 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table1); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,590 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table0 2020-07-20 23:53:17,591 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 2 regions. 2020-07-20 23:53:17,591 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table0); total cost=15.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,591 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table9 2020-07-20 23:53:17,592 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,592 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table9); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,592 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table8 2020-07-20 23:53:17,593 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,593 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table8); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,593 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table7 2020-07-20 23:53:17,594 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,594 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table7); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,594 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table6 2020-07-20 23:53:17,595 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,595 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table6); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,854 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table1380 2020-07-20 23:53:17,855 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,856 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table1380); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,856 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table1381 2020-07-20 23:53:17,857 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,858 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table1381); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,858 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table880 2020-07-20 23:53:17,859 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,859 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table880); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,859 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table1140 2020-07-20 23:53:17,860 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,861 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table1140); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,861 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table1382 2020-07-20 23:53:17,862 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,862 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table1382); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,862 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table881 2020-07-20 23:53:17,863 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,863 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table881); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,864 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table1141 2020-07-20 23:53:17,865 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,865 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table1141); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,866 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table1383 2020-07-20 23:53:17,866 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,867 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table1383); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,867 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table640 2020-07-20 23:53:17,868 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,868 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table640); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,869 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table882 2020-07-20 23:53:17,869 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,870 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table882); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,870 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table1142 2020-07-20 23:53:17,871 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,871 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table1142); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,871 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table1384 2020-07-20 23:53:17,872 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,873 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table1384); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,873 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table641 2020-07-20 23:53:17,874 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,874 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table641); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,874 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table883 2020-07-20 23:53:17,875 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,876 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table883); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,876 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table1143 2020-07-20 23:53:17,876 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,877 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table1143); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,877 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table1385 2020-07-20 23:53:17,878 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,878 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table1385); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,878 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table400 2020-07-20 23:53:17,879 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 2 regions. 2020-07-20 23:53:17,879 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table400); total cost=15.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,880 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table642 2020-07-20 23:53:17,880 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,881 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table642); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,881 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table884 2020-07-20 23:53:17,882 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,882 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table884); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,882 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table1144 2020-07-20 23:53:17,883 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,883 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table1144); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,883 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table1386 2020-07-20 23:53:17,884 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,884 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table1386); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,885 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table401 2020-07-20 23:53:17,885 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 2 regions. 2020-07-20 23:53:17,886 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table401); total cost=15.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,886 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table643 2020-07-20 23:53:17,886 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,887 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table643); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,887 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table885 2020-07-20 23:53:17,887 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,888 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table885); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,888 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table1145 2020-07-20 23:53:17,889 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,889 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table1145); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,889 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table1387 2020-07-20 23:53:17,890 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,890 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table1387); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,891 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table402 2020-07-20 23:53:17,891 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 2 regions. 2020-07-20 23:53:17,892 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table402); total cost=15.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,892 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table644 2020-07-20 23:53:17,892 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,893 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table644); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,893 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table886 2020-07-20 23:53:17,894 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,894 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table886); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,894 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table403 2020-07-20 23:53:17,895 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 2 regions. 2020-07-20 23:53:17,895 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table403); total cost=15.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,895 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table645 2020-07-20 23:53:17,896 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,896 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table645); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,897 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table887 2020-07-20 23:53:17,897 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,898 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table887); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,898 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table404 2020-07-20 23:53:17,899 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 2 regions. 2020-07-20 23:53:17,899 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table404); total cost=15.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,899 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table646 2020-07-20 23:53:17,900 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,900 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table646); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,901 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table888 2020-07-20 23:53:17,901 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,901 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table888); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,902 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table405 2020-07-20 23:53:17,902 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 2 regions. 2020-07-20 23:53:17,903 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table405); total cost=15.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,903 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table647 2020-07-20 23:53:17,904 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,904 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table647); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,904 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table889 2020-07-20 23:53:17,905 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,905 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table889); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,906 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table406 2020-07-20 23:53:17,907 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 2 regions. 2020-07-20 23:53:17,907 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table406); total cost=15.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,907 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table648 2020-07-20 23:53:17,908 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,908 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table648); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,909 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table407 2020-07-20 23:53:17,909 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 2 regions. 2020-07-20 23:53:17,910 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table407); total cost=15.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,910 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table649 2020-07-20 23:53:17,910 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,911 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table649); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,911 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table408 2020-07-20 23:53:17,911 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 2 regions. 2020-07-20 23:53:17,912 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table408); total cost=15.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,912 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table409 2020-07-20 23:53:17,913 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 2 regions. 2020-07-20 23:53:17,913 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table409); total cost=15.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,913 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table1146 2020-07-20 23:53:17,914 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,914 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table1146); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,914 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table1388 2020-07-20 23:53:17,915 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,915 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table1388); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,915 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table1147 2020-07-20 23:53:17,916 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,916 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table1147); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,916 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table1389 2020-07-20 23:53:17,917 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,917 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table1389); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,917 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table1148 2020-07-20 23:53:17,918 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,918 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table1148); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,918 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table1149 2020-07-20 23:53:17,918 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,919 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table1149); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,919 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table1370 2020-07-20 23:53:17,919 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,919 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table1370); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,920 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table1371 2020-07-20 23:53:17,920 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,920 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table1371); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,920 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table870 2020-07-20 23:53:17,921 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,921 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table870); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,922 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table1130 2020-07-20 23:53:17,922 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,922 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table1130); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,923 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table1372 2020-07-20 23:53:17,923 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,923 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table1372); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,924 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table871 2020-07-20 23:53:17,924 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,924 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table871); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,924 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table1131 2020-07-20 23:53:17,925 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,925 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table1131); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,925 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table1373 2020-07-20 23:53:17,925 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,926 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table1373); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,926 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table630 2020-07-20 23:53:17,926 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,926 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table630); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,927 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table872 2020-07-20 23:53:17,927 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,927 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table872); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,927 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table1132 2020-07-20 23:53:17,928 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,928 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table1132); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,928 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table1374 2020-07-20 23:53:17,929 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,929 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table1374); total cost=50.0, sum multiplier=570.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:17,929 INFO [Time-limited test] balancer.BaseLoadBalancer(1681): Start Generate Balance plan for table: table631 2020-07-20 23:53:17,929 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:17,930 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced table (table631); total cos ...[truncated 2959566 bytes]... :35,117 INFO [Time-limited test] balancer.StochasticLoadBalancer(432): start StochasticLoadBalancer.balancer, initCost=182.98605076606447, functionCost=RegionCountSkewCostFunction : (500.0, 0.31397210153212896); PrimaryRegionCountSkewCostFunction : (500.0, 0.0); MoveCostFunction : (7.0, 0.0); ServerLocalityCostFunction : (0.0, 1.0); RackLocalityCostFunction : (15.0, 1.0); TableSkewCostFunction : (35.0, 0.3142857142857143); RegionReplicaHostCostFunction : (100000.0, 0.0); RegionReplicaRackCostFunction : (10000.0, 0.0); ReadRequestCostFunction : (5.0, 0.0); WriteRequestCostFunction : (5.0, 0.0); MemStoreSizeCostFunction : (5.0, 0.0); StoreFileCostFunction : (5.0, 0.0); computedMaxSteps: 1000000 2020-07-20 23:53:51,146 INFO [Time-limited test] balancer.StochasticLoadBalancer(478): Finished computing new load balance plan. Computation took PT16.029S to try 1000000 different iterations. Found a solution that moves 968 regions; Going from a computed cost of 182.98605076606447 to a new cost of 28.581333333333333 2020-07-20 23:53:51,148 INFO [Time-limited test] balancer.BaseLoadBalancer(1689): Start Generate Balance plan for cluster. 2020-07-20 23:53:51,149 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 15 servers and 120 regions. 2020-07-20 23:53:51,149 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced cluster; total cost=142.1875, sum multiplier=577.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:51,149 INFO [Time-limited test] balancer.BaseLoadBalancer(1689): Start Generate Balance plan for cluster. 2020-07-20 23:53:51,149 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 10 servers and 10 regions. 2020-07-20 23:53:51,150 INFO [Time-limited test] balancer.StochasticLoadBalancer(432): start StochasticLoadBalancer.balancer, initCost=550.0, functionCost=RegionCountSkewCostFunction : (500.0, 1.0); PrimaryRegionCountSkewCostFunction : (500.0, 0.0); MoveCostFunction : (7.0, 0.0); ServerLocalityCostFunction : (0.0, 1.0); RackLocalityCostFunction : (15.0, 1.0); TableSkewCostFunction : (35.0, 1.0); RegionReplicaHostCostFunction : (100000.0, 0.0); RegionReplicaRackCostFunction : (10000.0, 0.0); ReadRequestCostFunction : (5.0, 0.0); WriteRequestCostFunction : (5.0, 0.0); MemStoreSizeCostFunction : (5.0, 0.0); StoreFileCostFunction : (5.0, 0.0); computedMaxSteps: 80000 2020-07-20 23:53:51,318 INFO [Time-limited test] balancer.StochasticLoadBalancer(478): Finished computing new load balance plan. Computation took PT0.169S to try 80000 different iterations. Found a solution that moves 9 regions; Going from a computed cost of 550.0 to a new cost of 56.3 2020-07-20 23:53:51,319 INFO [Time-limited test] balancer.BaseLoadBalancer(1689): Start Generate Balance plan for cluster. 2020-07-20 23:53:51,319 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 10 servers and 54 regions. 2020-07-20 23:53:51,319 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced cluster; total cost=41.406124739458065, sum multiplier=577.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:51,320 INFO [Time-limited test] balancer.BaseLoadBalancer(1689): Start Generate Balance plan for cluster. 2020-07-20 23:53:51,321 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 10 servers and 54 regions. 2020-07-20 23:53:51,321 INFO [Time-limited test] balancer.StochasticLoadBalancer(432): start StochasticLoadBalancer.balancer, initCost=550.0, functionCost=RegionCountSkewCostFunction : (500.0, 1.0); PrimaryRegionCountSkewCostFunction : (500.0, 0.0); MoveCostFunction : (7.0, 0.0); ServerLocalityCostFunction : (0.0, 1.0); RackLocalityCostFunction : (15.0, 1.0); TableSkewCostFunction : (35.0, 1.0); RegionReplicaHostCostFunction : (100000.0, 0.0); RegionReplicaRackCostFunction : (10000.0, 0.0); ReadRequestCostFunction : (5.0, 0.0); WriteRequestCostFunction : (5.0, 0.0); MemStoreSizeCostFunction : (5.0, 0.0); StoreFileCostFunction : (5.0, 0.0); computedMaxSteps: 432000 2020-07-20 23:53:52,389 INFO [Time-limited test] balancer.StochasticLoadBalancer(478): Finished computing new load balance plan. Computation took PT1.068S to try 432000 different iterations. Found a solution that moves 48 regions; Going from a computed cost of 550.0 to a new cost of 56.22222222222222 2020-07-20 23:53:52,390 INFO [Time-limited test] balancer.BaseLoadBalancer(1689): Start Generate Balance plan for cluster. 2020-07-20 23:53:52,391 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 10 servers and 55 regions. 2020-07-20 23:53:52,391 INFO [Time-limited test] balancer.StochasticLoadBalancer(432): start StochasticLoadBalancer.balancer, initCost=550.0, functionCost=RegionCountSkewCostFunction : (500.0, 1.0); PrimaryRegionCountSkewCostFunction : (500.0, 0.0); MoveCostFunction : (7.0, 0.0); ServerLocalityCostFunction : (0.0, 1.0); RackLocalityCostFunction : (15.0, 1.0); TableSkewCostFunction : (35.0, 1.0); RegionReplicaHostCostFunction : (100000.0, 0.0); RegionReplicaRackCostFunction : (10000.0, 0.0); ReadRequestCostFunction : (5.0, 0.0); WriteRequestCostFunction : (5.0, 0.0); MemStoreSizeCostFunction : (5.0, 0.0); StoreFileCostFunction : (5.0, 0.0); computedMaxSteps: 440000 2020-07-20 23:53:53,544 INFO [Time-limited test] balancer.StochasticLoadBalancer(478): Finished computing new load balance plan. Computation took PT1.153S to try 440000 different iterations. Found a solution that moves 49 regions; Going from a computed cost of 550.0 to a new cost of 56.236363636363635 2020-07-20 23:53:53,545 INFO [Time-limited test] balancer.BaseLoadBalancer(1689): Start Generate Balance plan for cluster. 2020-07-20 23:53:53,546 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 10 servers and 56 regions. 2020-07-20 23:53:53,546 INFO [Time-limited test] balancer.StochasticLoadBalancer(432): start StochasticLoadBalancer.balancer, initCost=550.0, functionCost=RegionCountSkewCostFunction : (500.0, 1.0); PrimaryRegionCountSkewCostFunction : (500.0, 0.0); MoveCostFunction : (7.0, 0.0); ServerLocalityCostFunction : (0.0, 1.0); RackLocalityCostFunction : (15.0, 1.0); TableSkewCostFunction : (35.0, 1.0); RegionReplicaHostCostFunction : (100000.0, 0.0); RegionReplicaRackCostFunction : (10000.0, 0.0); ReadRequestCostFunction : (5.0, 0.0); WriteRequestCostFunction : (5.0, 0.0); MemStoreSizeCostFunction : (5.0, 0.0); StoreFileCostFunction : (5.0, 0.0); computedMaxSteps: 448000 2020-07-20 23:53:54,624 INFO [Time-limited test] balancer.StochasticLoadBalancer(478): Finished computing new load balance plan. Computation took PT1.078S to try 448000 different iterations. Found a solution that moves 50 regions; Going from a computed cost of 550.0 to a new cost of 56.25 2020-07-20 23:53:54,624 INFO [Time-limited test] balancer.BaseLoadBalancer(1689): Start Generate Balance plan for cluster. 2020-07-20 23:53:54,625 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 10 servers and 16 regions. 2020-07-20 23:53:54,625 INFO [Time-limited test] balancer.StochasticLoadBalancer(432): start StochasticLoadBalancer.balancer, initCost=550.0, functionCost=RegionCountSkewCostFunction : (500.0, 1.0); PrimaryRegionCountSkewCostFunction : (500.0, 0.0); MoveCostFunction : (7.0, 0.0); ServerLocalityCostFunction : (0.0, 1.0); RackLocalityCostFunction : (15.0, 1.0); TableSkewCostFunction : (35.0, 1.0); RegionReplicaHostCostFunction : (100000.0, 0.0); RegionReplicaRackCostFunction : (10000.0, 0.0); ReadRequestCostFunction : (5.0, 0.0); WriteRequestCostFunction : (5.0, 0.0); MemStoreSizeCostFunction : (5.0, 0.0); StoreFileCostFunction : (5.0, 0.0); computedMaxSteps: 128000 2020-07-20 23:53:54,885 INFO [Time-limited test] balancer.StochasticLoadBalancer(478): Finished computing new load balance plan. Computation took PT0.26S to try 128000 different iterations. Found a solution that moves 14 regions; Going from a computed cost of 550.0 to a new cost of 56.125 2020-07-20 23:53:54,886 INFO [Time-limited test] balancer.BaseLoadBalancer(1689): Start Generate Balance plan for cluster. 2020-07-20 23:53:54,886 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 10 servers and 17 regions. 2020-07-20 23:53:54,886 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced cluster; total cost=188.50267379679147, sum multiplier=577.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:54,886 INFO [Time-limited test] balancer.BaseLoadBalancer(1689): Start Generate Balance plan for cluster. 2020-07-20 23:53:54,886 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 10 servers and 18 regions. 2020-07-20 23:53:54,886 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced cluster; total cost=222.33637747336377, sum multiplier=577.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:54,887 INFO [Time-limited test] balancer.BaseLoadBalancer(1689): Start Generate Balance plan for cluster. 2020-07-20 23:53:54,887 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 10 servers and 19 regions. 2020-07-20 23:53:54,887 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced cluster; total cost=253.80116959064318, sum multiplier=577.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:54,888 INFO [Time-limited test] balancer.BaseLoadBalancer(1689): Start Generate Balance plan for cluster. 2020-07-20 23:53:54,889 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 10 servers and 132 regions. 2020-07-20 23:53:54,889 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced cluster; total cost=508.9525804116246, sum multiplier=577.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:54,890 INFO [Time-limited test] balancer.BaseLoadBalancer(1689): Start Generate Balance plan for cluster. 2020-07-20 23:53:54,891 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 10 servers and 164 regions. 2020-07-20 23:53:54,891 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced cluster; total cost=516.8741181213464, sum multiplier=577.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:54,891 INFO [Time-limited test] balancer.BaseLoadBalancer(1689): Start Generate Balance plan for cluster. 2020-07-20 23:53:54,892 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 8 servers and 81 regions. 2020-07-20 23:53:54,892 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced cluster; total cost=57.98500881834215, sum multiplier=577.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:54,892 INFO [Time-limited test] balancer.BaseLoadBalancer(1689): Start Generate Balance plan for cluster. 2020-07-20 23:53:54,893 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 8 servers and 81 regions. 2020-07-20 23:53:54,893 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced cluster; total cost=58.87786596119929, sum multiplier=577.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:54,894 INFO [Time-limited test] balancer.BaseLoadBalancer(1689): Start Generate Balance plan for cluster. 2020-07-20 23:53:54,896 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 8 servers and 414 regions. 2020-07-20 23:53:54,896 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced cluster; total cost=246.39961031265378, sum multiplier=577.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:54,898 INFO [Time-limited test] balancer.BaseLoadBalancer(1689): Start Generate Balance plan for cluster. 2020-07-20 23:53:54,901 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 8 servers and 810 regions. 2020-07-20 23:53:54,901 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced cluster; total cost=63.5731772776665, sum multiplier=577.0; cost/multiplier to need a balance is 1.0 2020-07-20 23:53:54,901 INFO [Time-limited test] balancer.BaseLoadBalancer(1689): Start Generate Balance plan for cluster. 2020-07-20 23:53:54,901 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 5 servers and 20 regions. 2020-07-20 23:53:54,901 INFO [Time-limited test] balancer.StochasticLoadBalancer(432): start StochasticLoadBalancer.balancer, initCost=142.1875, functionCost=RegionCountSkewCostFunction : (500.0, 0.25); PrimaryRegionCountSkewCostFunction : (500.0, 0.0); MoveCostFunction : (7.0, 0.0); ServerLocalityCostFunction : (0.0, 1.0); RackLocalityCostFunction : (15.0, 1.0); TableSkewCostFunction : (35.0, 0.0625); RegionReplicaHostCostFunction : (100000.0, 0.0); RegionReplicaRackCostFunction : (10000.0, 0.0); ReadRequestCostFunction : (5.0, 0.0); WriteRequestCostFunction : (5.0, 0.0); MemStoreSizeCostFunction : (5.0, 0.0); StoreFileCostFunction : (5.0, 0.0); computedMaxSteps: 80000 2020-07-20 23:53:55,034 INFO [Time-limited test] balancer.StochasticLoadBalancer(478): Finished computing new load balance plan. Computation took PT0.133S to try 80000 different iterations. Found a solution that moves 4 regions; Going from a computed cost of 142.1875 to a new cost of 18.5875 2020-07-20 23:53:55,057 INFO [Time-limited test] balancer.BaseLoadBalancer(1689): Start Generate Balance plan for cluster. 2020-07-20 23:53:55,067 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 393 servers and 56 regions. 2020-07-20 23:53:55,068 WARN [Time-limited test] balancer.StochasticLoadBalancer(425): calculatedMaxSteps:17606400 for loadbalancer's stochastic walk is larger than maxSteps:1000000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2020-07-20 23:53:55,068 INFO [Time-limited test] balancer.StochasticLoadBalancer(432): start StochasticLoadBalancer.balancer, initCost=549.9999999999674, functionCost=RegionCountSkewCostFunction : (500.0, 0.9999999999999347); PrimaryRegionCountSkewCostFunction : (500.0, 0.0); MoveCostFunction : (7.0, 0.0); ServerLocalityCostFunction : (0.0, 1.0); RackLocalityCostFunction : (15.0, 1.0); TableSkewCostFunction : (35.0, 1.0); RegionReplicaHostCostFunction : (100000.0, 0.0); RegionReplicaRackCostFunction : (10000.0, 0.0); ReadRequestCostFunction : (5.0, 0.0); WriteRequestCostFunction : (5.0, 0.0); MemStoreSizeCostFunction : (5.0, 0.0); StoreFileCostFunction : (5.0, 0.0); computedMaxSteps: 1000000 2020-07-20 23:53:57,375 INFO [Time-limited test] balancer.StochasticLoadBalancer(478): Finished computing new load balance plan. Computation took PT2.308S to try 1000000 different iterations. Found a solution that moves 55 regions; Going from a computed cost of 549.9999999999674 to a new cost of 56.875 2020-07-20 23:53:57,377 INFO [Time-limited test] balancer.BaseLoadBalancer(1053): slop=0.0, systemTablesOnMaster=false 2020-07-20 23:53:57,377 INFO [Time-limited test] balancer.StochasticLoadBalancer(224): Loaded config; maxSteps=1000000, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] etc. 2020-07-20 23:53:57,387 INFO [Time-limited test] hbase.ResourceChecker(179): after: master.balancer.TestStochasticLoadBalancer#testNeedBalance Thread=10 (was 10), OpenFileDescriptor=216 (was 216), MaxFileDescriptor=1048576 (was 1048576), SystemLoadAverage=546 (was 637), ProcessCount=14 (was 14), AvailableMemoryMB=11074 (was 12094) 2020-07-20 23:53:57,395 INFO [Time-limited test] hbase.ResourceChecker(151): before: master.balancer.TestStochasticLoadBalancer#testRegionLoadCost Thread=10, OpenFileDescriptor=216, MaxFileDescriptor=1048576, SystemLoadAverage=546, ProcessCount=14, AvailableMemoryMB=11073 2020-07-20 23:53:58,054 INFO [Time-limited test] hbase.ResourceChecker(179): after: master.balancer.TestStochasticLoadBalancer#testRegionLoadCost Thread=10 (was 10), OpenFileDescriptor=216 (was 216), MaxFileDescriptor=1048576 (was 1048576), SystemLoadAverage=546 (was 546), ProcessCount=14 (was 14), AvailableMemoryMB=11056 (was 11073) 2020-07-20 23:53:58,060 INFO [Time-limited test] hbase.ResourceChecker(151): before: master.balancer.TestStochasticLoadBalancer#testSkewCost Thread=10, OpenFileDescriptor=216, MaxFileDescriptor=1048576, SystemLoadAverage=546, ProcessCount=14, AvailableMemoryMB=11059 2020-07-20 23:53:58,070 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 1 servers and 0 regions. 2020-07-20 23:53:58,070 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 1 servers and 1 regions. 2020-07-20 23:53:58,070 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 1 servers and 10 regions. 2020-07-20 23:53:58,070 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 0 regions. 2020-07-20 23:53:58,070 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 2 regions. 2020-07-20 23:53:58,071 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 3 regions. 2020-07-20 23:53:58,071 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 4 regions. 2020-07-20 23:53:58,071 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 5 regions. 2020-07-20 23:53:58,071 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 6 regions. 2020-07-20 23:53:58,071 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 2 regions. 2020-07-20 23:53:58,071 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:53:58,072 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 11 regions. 2020-07-20 23:53:58,090 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1946 regions. 2020-07-20 23:53:58,090 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 101 regions. 2020-07-20 23:53:58,090 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 3 servers and 3 regions. 2020-07-20 23:53:58,091 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 3 servers and 6 regions. 2020-07-20 23:53:58,091 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 3 servers and 4 regions. 2020-07-20 23:53:58,091 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 3 servers and 3 regions. 2020-07-20 23:53:58,091 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 3 servers and 4 regions. 2020-07-20 23:53:58,092 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 3 servers and 40 regions. 2020-07-20 23:53:58,092 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 4 servers and 6 regions. 2020-07-20 23:53:58,092 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 4 servers and 4 regions. 2020-07-20 23:53:58,092 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 4 servers and 5 regions. 2020-07-20 23:53:58,093 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 4 servers and 12 regions. 2020-07-20 23:53:58,093 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 4 servers and 8 regions. 2020-07-20 23:53:58,093 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 4 servers and 7 regions. 2020-07-20 23:53:58,093 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 4 servers and 6 regions. 2020-07-20 23:53:58,094 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 4 servers and 19 regions. 2020-07-20 23:53:58,094 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 4 servers and 20 regions. 2020-07-20 23:53:58,095 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 4 servers and 7 regions. 2020-07-20 23:53:58,095 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 5 servers and 8 regions. 2020-07-20 23:53:58,126 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 6 servers and 3010 regions. 2020-07-20 23:53:58,161 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 6 servers and 3500 regions. 2020-07-20 23:53:58,162 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 15 servers and 120 regions. 2020-07-20 23:53:58,162 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 10 servers and 10 regions. 2020-07-20 23:53:58,163 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 10 servers and 54 regions. 2020-07-20 23:53:58,163 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 10 servers and 54 regions. 2020-07-20 23:53:58,164 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 10 servers and 55 regions. 2020-07-20 23:53:58,164 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 10 servers and 56 regions. 2020-07-20 23:53:58,164 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 10 servers and 16 regions. 2020-07-20 23:53:58,165 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 10 servers and 17 regions. 2020-07-20 23:53:58,165 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 10 servers and 18 regions. 2020-07-20 23:53:58,165 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 10 servers and 19 regions. 2020-07-20 23:53:58,166 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 10 servers and 132 regions. 2020-07-20 23:53:58,167 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 10 servers and 164 regions. 2020-07-20 23:53:58,168 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 8 servers and 81 regions. 2020-07-20 23:53:58,168 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 8 servers and 81 regions. 2020-07-20 23:53:58,169 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 8 servers and 414 regions. 2020-07-20 23:53:58,171 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 8 servers and 810 regions. 2020-07-20 23:53:58,171 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 5 servers and 20 regions. 2020-07-20 23:53:58,176 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 393 servers and 56 regions. 2020-07-20 23:53:58,177 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 5 servers and 1 regions. 2020-07-20 23:53:58,177 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 5 servers and 2 regions. 2020-07-20 23:53:58,177 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 5 servers and 3 regions. 2020-07-20 23:53:58,177 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 5 servers and 4 regions. 2020-07-20 23:53:58,178 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 5 servers and 5 regions. 2020-07-20 23:54:01,594 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 5 servers and 10000 regions. 2020-07-20 23:54:01,601 INFO [Time-limited test] hbase.ResourceChecker(179): after: master.balancer.TestStochasticLoadBalancer#testSkewCost Thread=10 (was 10), OpenFileDescriptor=216 (was 216), MaxFileDescriptor=1048576 (was 1048576), SystemLoadAverage=546 (was 546), ProcessCount=14 (was 14), AvailableMemoryMB=11944 (was 11059) - AvailableMemoryMB LEAK? - 2020-07-20 23:54:01,607 INFO [Time-limited test] hbase.ResourceChecker(151): before: master.balancer.TestStochasticLoadBalancer#testLosingRs Thread=10, OpenFileDescriptor=217, MaxFileDescriptor=1048576, SystemLoadAverage=546, ProcessCount=14, AvailableMemoryMB=11943 2020-07-20 23:54:01,608 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 3 servers and 20 regions. 2020-07-20 23:54:01,609 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): We need to load balance cluster; total cost=297.40789473684214, sum multiplier=570.0; cost/multiplier to need a balance is 0.05 2020-07-20 23:54:01,609 INFO [Time-limited test] balancer.StochasticLoadBalancer(432): start StochasticLoadBalancer.balancer, initCost=297.40789473684214, functionCost=RegionCountSkewCostFunction : (500.0, 0.5263157894736843); PrimaryRegionCountSkewCostFunction : (500.0, 0.0); MoveCostFunction : (0.0, 0.0); ServerLocalityCostFunction : (0.0, 1.0); RackLocalityCostFunction : (15.0, 1.0); TableSkewCostFunction : (35.0, 0.55); RegionReplicaHostCostFunction : (100000.0, 0.0); RegionReplicaRackCostFunction : (10000.0, 0.0); ReadRequestCostFunction : (5.0, 0.0); WriteRequestCostFunction : (5.0, 0.0); MemStoreSizeCostFunction : (5.0, 0.0); StoreFileCostFunction : (5.0, 0.0); computedMaxSteps: 48000 2020-07-20 23:54:01,715 INFO [Time-limited test] balancer.StochasticLoadBalancer(478): Finished computing new load balance plan. Computation took PT0.107S to try 48000 different iterations. Found a solution that moves 7 regions; Going from a computed cost of 297.40789473684214 to a new cost of 18.325 2020-07-20 23:54:01,716 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 3 servers and 20 regions. 2020-07-20 23:54:01,716 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(350): Skipping load balancing because balanced cluster; total cost=15.875, sum multiplier=577.0; cost/multiplier to need a balance is 0.05 2020-07-20 23:54:01,723 INFO [Time-limited test] hbase.ResourceChecker(179): after: master.balancer.TestStochasticLoadBalancer#testLosingRs Thread=10 (was 10), OpenFileDescriptor=216 (was 217), MaxFileDescriptor=1048576 (was 1048576), SystemLoadAverage=546 (was 546), ProcessCount=14 (was 14), AvailableMemoryMB=11933 (was 11943) 2020-07-20 23:54:01,728 INFO [Time-limited test] hbase.ResourceChecker(151): before: master.balancer.TestStochasticLoadBalancer#testCostAfterUndoAction Thread=10, OpenFileDescriptor=216, MaxFileDescriptor=1048576, SystemLoadAverage=546, ProcessCount=14, AvailableMemoryMB=11933 2020-07-20 23:54:01,729 INFO [Time-limited test] balancer.BaseLoadBalancer(1053): slop=0.0, systemTablesOnMaster=false 2020-07-20 23:54:01,729 INFO [Time-limited test] balancer.StochasticLoadBalancer(224): Loaded config; maxSteps=1000000, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] etc. 2020-07-20 23:54:01,729 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 1 servers and 0 regions. 2020-07-20 23:54:01,730 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 1 servers and 1 regions. 2020-07-20 23:54:01,730 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 1 servers and 10 regions. 2020-07-20 23:54:01,731 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 0 regions. 2020-07-20 23:54:01,731 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 2 regions. 2020-07-20 23:54:01,731 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 3 regions. 2020-07-20 23:54:01,731 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 4 regions. 2020-07-20 23:54:01,732 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 5 regions. 2020-07-20 23:54:01,732 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 6 regions. 2020-07-20 23:54:01,732 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 2 regions. 2020-07-20 23:54:01,732 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1 regions. 2020-07-20 23:54:01,732 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 11 regions. 2020-07-20 23:54:01,755 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 1946 regions. 2020-07-20 23:54:01,757 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 2 servers and 101 regions. 2020-07-20 23:54:01,757 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 3 servers and 3 regions. 2020-07-20 23:54:01,757 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 3 servers and 6 regions. 2020-07-20 23:54:01,758 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 3 servers and 4 regions. 2020-07-20 23:54:01,758 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 3 servers and 3 regions. 2020-07-20 23:54:01,758 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 3 servers and 4 regions. 2020-07-20 23:54:01,759 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 3 servers and 40 regions. 2020-07-20 23:54:01,759 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 4 servers and 6 regions. 2020-07-20 23:54:01,759 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 4 servers and 4 regions. 2020-07-20 23:54:01,759 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 4 servers and 5 regions. 2020-07-20 23:54:01,760 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 4 servers and 12 regions. 2020-07-20 23:54:01,760 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 4 servers and 8 regions. 2020-07-20 23:54:01,761 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 4 servers and 7 regions. 2020-07-20 23:54:01,761 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 4 servers and 6 regions. 2020-07-20 23:54:01,761 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 4 servers and 19 regions. 2020-07-20 23:54:01,762 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 4 servers and 20 regions. 2020-07-20 23:54:01,762 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 4 servers and 7 regions. 2020-07-20 23:54:01,762 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 5 servers and 8 regions. 2020-07-20 23:54:01,795 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 6 servers and 3010 regions. 2020-07-20 23:54:01,831 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 6 servers and 3500 regions. 2020-07-20 23:54:01,833 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 15 servers and 120 regions. 2020-07-20 23:54:01,833 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 10 servers and 10 regions. 2020-07-20 23:54:01,834 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 10 servers and 54 regions. 2020-07-20 23:54:01,835 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 10 servers and 54 regions. 2020-07-20 23:54:01,835 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 10 servers and 55 regions. 2020-07-20 23:54:01,836 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 10 servers and 56 regions. 2020-07-20 23:54:01,836 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 10 servers and 16 regions. 2020-07-20 23:54:01,836 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 10 servers and 17 regions. 2020-07-20 23:54:01,837 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 10 servers and 18 regions. 2020-07-20 23:54:01,837 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 10 servers and 19 regions. 2020-07-20 23:54:01,838 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 10 servers and 132 regions. 2020-07-20 23:54:01,839 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 10 servers and 164 regions. 2020-07-20 23:54:01,840 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 8 servers and 81 regions. 2020-07-20 23:54:01,840 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 8 servers and 81 regions. 2020-07-20 23:54:01,842 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 8 servers and 414 regions. 2020-07-20 23:54:01,846 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 8 servers and 810 regions. 2020-07-20 23:54:01,846 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 5 servers and 20 regions. 2020-07-20 23:54:01,851 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(894): RegionCountSkewCostFunction sees a total of 393 servers and 56 regions. 2020-07-20 23:54:01,858 INFO [Time-limited test] hbase.ResourceChecker(179): after: master.balancer.TestStochasticLoadBalancer#testCostAfterUndoAction Thread=10 (was 10), OpenFileDescriptor=216 (was 216), MaxFileDescriptor=1048576 (was 1048576), SystemLoadAverage=590 (was 546) - SystemLoadAverage LEAK? -, ProcessCount=14 (was 14), AvailableMemoryMB=11927 (was 11933) 2020-07-20 23:54:01,864 INFO [Time-limited test] hbase.ResourceChecker(151): before: master.balancer.TestStochasticLoadBalancer#testMoveCostMultiplier Thread=10, OpenFileDescriptor=216, MaxFileDescriptor=1048576, SystemLoadAverage=590, ProcessCount=14, AvailableMemoryMB=11926 2020-07-20 23:54:01,891 INFO [Time-limited test] hbase.ResourceChecker(179): after: master.balancer.TestStochasticLoadBalancer#testMoveCostMultiplier Thread=10 (was 10), OpenFileDescriptor=216 (was 216), MaxFileDescriptor=1048576 (was 1048576), SystemLoadAverage=590 (was 590), ProcessCount=14 (was 14), AvailableMemoryMB=11925 (was 11926) 2020-07-20 23:54:01,898 INFO [Time-limited test] hbase.ResourceChecker(151): before: master.balancer.TestStochasticLoadBalancer#testLocalityCost Thread=10, OpenFileDescriptor=216, MaxFileDescriptor=1048576, SystemLoadAverage=590, ProcessCount=14, AvailableMemoryMB=11924 2020-07-20 23:54:01,939 INFO [Time-limited test] hbase.ResourceChecker(179): after: master.balancer.TestStochasticLoadBalancer#testLocalityCost Thread=10 (was 10), OpenFileDescriptor=216 (was 216), MaxFileDescriptor=1048576 (was 1048576), SystemLoadAverage=590 (was 590), ProcessCount=14 (was 14), AvailableMemoryMB=11920 (was 11924) 2020-07-20 23:54:01,947 INFO [Time-limited test] hbase.ResourceChecker(151): before: master.balancer.TestStochasticLoadBalancer#testCostFromArray Thread=10, OpenFileDescriptor=216, MaxFileDescriptor=1048576, SystemLoadAverage=590, ProcessCount=14, AvailableMemoryMB=11920 2020-07-20 23:54:01,971 INFO [Time-limited test] hbase.ResourceChecker(179): after: master.balancer.TestStochasticLoadBalancer#testCostFromArray Thread=10 (was 10), OpenFileDescriptor=216 (was 216), MaxFileDescriptor=1048576 (was 1048576), SystemLoadAverage=590 (was 590), ProcessCount=14 (was 14), AvailableMemoryMB=11918 (was 11920) 2020-07-20 23:54:01,978 INFO [Time-limited test] hbase.ResourceChecker(151): before: master.balancer.TestStochasticLoadBalancer#testMoveCost Thread=10, OpenFileDescriptor=216, MaxFileDescriptor=1048576, SystemLoadAverage=590, ProcessCount=14, AvailableMemoryMB=11917 2020-07-20 23:54:02,138 INFO [Time-limited test] hbase.ResourceChecker(179): after: master.balancer.TestStochasticLoadBalancer#testMoveCost Thread=10 (was 10), OpenFileDescriptor=216 (was 216), MaxFileDescriptor=1048576 (was 1048576), SystemLoadAverage=590 (was 590), ProcessCount=14 (was 14), AvailableMemoryMB=11909 (was 11917) 2020-07-20 23:54:02,145 INFO [Time-limited test] hbase.ResourceChecker(151): before: master.balancer.TestStochasticLoadBalancer#testAdditionalCostFunction Thread=10, OpenFileDescriptor=216, MaxFileDescriptor=1048576, SystemLoadAverage=590, ProcessCount=14, AvailableMemoryMB=11909 2020-07-20 23:54:02,147 INFO [Time-limited test] balancer.BaseLoadBalancer(1053): slop=0.0, systemTablesOnMaster=false 2020-07-20 23:54:02,150 INFO [Time-limited test] balancer.StochasticLoadBalancer(249): Successfully loaded custom CostFunction 'DummyCostFunction' 2020-07-20 23:54:02,150 INFO [Time-limited test] balancer.StochasticLoadBalancer(224): Loaded config; maxSteps=1000000, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction, DummyCostFunction] etc. 2020-07-20 23:54:02,161 INFO [Time-limited test] hbase.ResourceChecker(179): after: master.balancer.TestStochasticLoadBalancer#testAdditionalCostFunction Thread=10 (was 10), OpenFileDescriptor=216 (was 216), MaxFileDescriptor=1048576 (was 1048576), SystemLoadAverage=590 (was 590), ProcessCount=14 (was 14), AvailableMemoryMB=11909 (was 11909) 2020-07-20 23:54:02,168 INFO [Time-limited test] hbase.ResourceChecker(151): before: master.balancer.TestStochasticLoadBalancer#testKeepRegionLoad Thread=10, OpenFileDescriptor=216, MaxFileDescriptor=1048576, SystemLoadAverage=590, ProcessCount=14, AvailableMemoryMB=11908 2020-07-20 23:54:13,575 INFO [Time-limited test] hbase.ResourceChecker(179): after: master.balancer.TestStochasticLoadBalancer#testKeepRegionLoad Thread=10 (was 10), OpenFileDescriptor=216 (was 216), MaxFileDescriptor=1048576 (was 1048576), SystemLoadAverage=576 (was 590), ProcessCount=14 (was 14), AvailableMemoryMB=11677 (was 11908) 2020-07-20 23:54:13,581 INFO [Time-limited test] hbase.ResourceChecker(151): before: master.balancer.TestStochasticLoadBalancer#testTableSkewCost Thread=10, OpenFileDescriptor=217, MaxFileDescriptor=1048576, SystemLoadAverage=576, ProcessCount=14, AvailableMemoryMB=11677 2020-07-20 23:54:13,661 INFO [Time-limited test] hbase.ResourceChecker(179): after: master.balancer.TestStochasticLoadBalancer#testTableSkewCost Thread=10 (was 10), OpenFileDescriptor=216 (was 217), MaxFileDescriptor=1048576 (was 1048576), SystemLoadAverage=576 (was 576), ProcessCount=14 (was 14), AvailableMemoryMB=11677 (was 11677) 2020-07-20 23:54:13,696 INFO [Time-limited test] balancer.BaseLoadBalancer(1053): slop=0.0, systemTablesOnMaster=false 2020-07-20 23:54:13,696 INFO [Time-limited test] balancer.StochasticLoadBalancer(224): Loaded config; maxSteps=1000000, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] etc. 2020-07-20 23:54:13,701 INFO [Time-limited test] hbase.ResourceChecker(151): before: master.balancer.TestStochasticLoadBalancer#testMoveCostMultiplier Thread=10, OpenFileDescriptor=216, MaxFileDescriptor=1048576, SystemLoadAverage=576, ProcessCount=14, AvailableMemoryMB=11679 2020-07-20 23:54:13,714 INFO [Time-limited test] hbase.ResourceChecker(179): after: master.balancer.TestStochasticLoadBalancer#testMoveCostMultiplier Thread=10 (was 10), OpenFileDescriptor=216 (was 216), MaxFileDescriptor=1048576 (was 1048576), SystemLoadAverage=576 (was 576), ProcessCount=14 (was 14), AvailableMemoryMB=11679 (was 11679) 2020-07-20 23:54:13,734 INFO [Time-limited test] balancer.BaseLoadBalancer(1053): slop=0.0, systemTablesOnMaster=false 2020-07-20 23:54:13,734 INFO [Time-limited test] balancer.StochasticLoadBalancer(224): Loaded config; maxSteps=1000000, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] etc. 2020-07-20 23:54:13,741 INFO [Time-limited test] hbase.ResourceChecker(151): before: master.balancer.TestStochasticLoadBalancer#testMoveCostMultiplier Thread=10, OpenFileDescriptor=216, MaxFileDescriptor=1048576, SystemLoadAverage=576, ProcessCount=14, AvailableMemoryMB=11679 2020-07-20 23:54:13,758 INFO [Time-limited test] hbase.ResourceChecker(179): after: master.balancer.TestStochasticLoadBalancer#testMoveCostMultiplier Thread=10 (was 10), OpenFileDescriptor=216 (was 216), MaxFileDescriptor=1048576 (was 1048576), SystemLoadAverage=576 (was 576), ProcessCount=14 (was 14), AvailableMemoryMB=11679 (was 11679) Standard Error 2020-07-20 23:54:01,858 INFO [Time-limited test] hbase.ResourceChecker(179): after: master.balancer.TestStochasticLoadBalancer#testCostAfterUndoAction Thread=10 (was 10), OpenFileDescriptor=216 (was 216), MaxFileDescriptor=1048576 (was 1048576), SystemLoadAverage=590 (was 546) - SystemLoadAverage LEAK? -, ProcessCount=14 (was 14), AvailableMemoryMB=11927 (was 11933) 2020-07-20 23:54:01,864 INFO [Time-limited test] hbase.ResourceChecker(151): before: master.balancer.TestStochasticLoadBalancer#testMoveCostMultiplier Thread=10, OpenFileDescriptor=216, MaxFileDescriptor=1048576, SystemLoadAverage=590, ProcessCount=14, AvailableMemoryMB=11926