[HBASE-22739] ArrayIndexOutOfBoundsException when balance

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0-alpha-1, 2.3.0, 2.1.8, 2.2.3
• Component/s: Balancer
• Hadoop Flags: Reviewed
• Assignee: Lijin Bin
• Reporter: casuallc
• Created: 26/Jul/19 03:22
• Updated: 26/Mar/21 06:42
• Resolved: 30/Oct/19 20:35

Description

 

2019-07-25 15:19:59,828 ERROR [master/nna:16000.Chore.1] hbase.ScheduledChore: Caught error
java.lang.ArrayIndexOutOfBoundsException: 3171
at org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.removeRegion(BaseLoadBalancer.java:873)
at org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.doAction(BaseLoadBalancer.java:716)
at org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer.balanceCluster(StochasticLoadBalancer.java:407)
at org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer.balanceCluster(StochasticLoadBalancer.java:318)
at org.apache.hadoop.hbase.master.HMaster.balance(HMaster.java:1650)
at org.apache.hadoop.hbase.master.HMaster.balance(HMaster.java:1567)
at org.apache.hadoop.hbase.master.balancer.BalancerChore.chore(BalancerChore.java:49)
at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:186)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:111)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

should check if the regionIndex is valid when removeRegion,

java: hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java

int[] removeRegion(int[] regions, int regionIndex) {
    //TODO: this maybe costly. Consider using linked lists
    int[] newRegions = new int[regions.length - 1];
    int i = 0;
    for (i = 0; i < regions.length; i++) {
        if (regions[i] == regionIndex) {
            break;
        }
        if (i == regions.length - 1) {
            return Arrays.copyOf(regions, regions.length);
        }
        newRegions[i] = regions[i];
    }
    System.arraycopy(regions, i+1, newRegions, i, newRegions.length - i);
    return newRegions;
}

Issue Links

• is duplicated by: HBASE-23173 ArrayIndexOutOfBoundsException in BaseLoadBalancer$Cluster.removeRegion
• links to: GitHub Pull Request #411
• links to: GitHub Pull Request #729