Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-25767

CandidateGenerator.getRandomIterationOrder is too slow on large cluster

    XMLWordPrintableJSON

    Details

    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      In the actual implementation classes of CandidateGenerator, now we just random select a start point and then iterate sequentially, instead of using the old way, where we will create a big array to hold all the integers in [0, num_regions_in_cluster), shuffle the array, and then iterate on the array.
      The new implementation is 'random' enough as every time we just select one candidate. The problem for the old implementation is that, it will create an array every time when we want to get a candidate, if we have tens of thousands regions, we will create an array with tens of thousands length everytime, which causes big GC pressure and slow down the balancer execution.
      Show
      In the actual implementation classes of CandidateGenerator, now we just random select a start point and then iterate sequentially, instead of using the old way, where we will create a big array to hold all the integers in [0, num_regions_in_cluster), shuffle the array, and then iterate on the array. The new implementation is 'random' enough as every time we just select one candidate. The problem for the old implementation is that, it will create an array every time when we want to get a candidate, if we have tens of thousands regions, we will create an array with tens of thousands length everytime, which causes big GC pressure and slow down the balancer execution.

      Description

      Similar to HBASE-25759, it is just used to test whether we should skip calculation, but in production masterServices will never be null.

      ==========

      Update, change the title of this issue for removing CandidateGenerator.getRandomIterationOrder as it is too slow which causes the CandidateGenerator.getRandomIterationOrder to fail when we remove the masterServices field in LocalityBasedCandidateGenerator. As this is the most important change in this issue so change the title of this issue.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                zhangduo Duo Zhang
                Reporter:
                zhangduo Duo Zhang
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: