HBase
  1. HBase
  2. HBASE-3943

Propagate compaction status from HRegion to HServerLoad.RegionLoad

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: master, regionserver
    • Labels:
      None

      Description

      Load balancer works with HRegionInfo. However, compaction status is contained in HRegion.WriteState

      Users reported the following (Schubert Zhang):

      1. A region have many files, the compacting takes long time.
      2. But the balancer (default 5 minutes) close and move this region to
      another server.
      3. Then, the compacting start again.
      4. Then, then balancer close and move it to another server.

      Thus, the compacting cannot complete.
      Now, we set the balancer interval to 30 minutes to remission this issue.

      We need to propagate compaction status to HServerLoad.RegionLoad so that load balancer doesn't move the region which is being compacted.

        Issue Links

          Activity

          Hide
          Ted Yu added a comment -

          First attempt at solving this issue.

          Show
          Ted Yu added a comment - First attempt at solving this issue.
          Hide
          stack added a comment -

          Are we mixing region schema and region metrics when we add this to HRI Ted? I'm not a fan of it. And do you think this the way to go? By the time the balancer goes to act on the info, it'll be stale, don't you think? It seems like the balancer issue that the boys were complaining off could be addressed by having the balancer not move regions it just moved in the previous balance run? Thanks Ted.

          Show
          stack added a comment - Are we mixing region schema and region metrics when we add this to HRI Ted? I'm not a fan of it. And do you think this the way to go? By the time the balancer goes to act on the info, it'll be stale, don't you think? It seems like the balancer issue that the boys were complaining off could be addressed by having the balancer not move regions it just moved in the previous balance run? Thanks Ted.
          Hide
          Ted Yu added a comment -

          Thanks for the reminder, Stack.
          HRI isn't a proper carrier for compaction status.
          HRegionServer.buildServerLoad() has a chance to pass compaction status to Master. But in trunk, load balancer doesn't have access to HServerLoad (including RegionLoad).

          I agree that if a region was reassigned in round N, it shouldn't be reassigned again in round N+1, ideally.
          I will log another issue for that.

          Show
          Ted Yu added a comment - Thanks for the reminder, Stack. HRI isn't a proper carrier for compaction status. HRegionServer.buildServerLoad() has a chance to pass compaction status to Master. But in trunk, load balancer doesn't have access to HServerLoad (including RegionLoad). I agree that if a region was reassigned in round N, it shouldn't be reassigned again in round N+1, ideally. I will log another issue for that.
          Hide
          Ted Yu added a comment -

          From Schubert:

          In my use case, the region-store size is aggresively set to be very large (such as 1GB, even 10GB), then, the time of compaction would take tens of minutes (e.g. 30 minutes).
           
          hbase.regionserver.msginterval=3000, it is the default value.
          

          The benefit of passing compaction status through RegionLoad is that load balancer wouldn't interrupt a half hour compaction 5 minutes after it starts (default value of hbase.balancer.period, some users make it longer to reduce impact of region reassignment).
          There is still a chance that compaction is interrupted 3 seconds after it starts, but the cost is much lower than the above scenario.

          Show
          Ted Yu added a comment - From Schubert: In my use case, the region-store size is aggresively set to be very large (such as 1GB, even 10GB), then, the time of compaction would take tens of minutes (e.g. 30 minutes). hbase.regionserver.msginterval=3000, it is the default value. The benefit of passing compaction status through RegionLoad is that load balancer wouldn't interrupt a half hour compaction 5 minutes after it starts (default value of hbase.balancer.period, some users make it longer to reduce impact of region reassignment). There is still a chance that compaction is interrupted 3 seconds after it starts, but the cost is much lower than the above scenario.
          Hide
          stack added a comment -

          While passing general outline of compaction status on regionserver might make sense giving a feel for what a cluster is up to, I do not think that it should be used as a factor balancing. For one the info the regionserver passes will likely be stale by the time the master goes to act on it (excepting the extreme case that Schubert outlines above). Better I think is the suggestion above where we do not move recently balanced regions. That should work for schubert's case.

          Show
          stack added a comment - While passing general outline of compaction status on regionserver might make sense giving a feel for what a cluster is up to, I do not think that it should be used as a factor balancing. For one the info the regionserver passes will likely be stale by the time the master goes to act on it (excepting the extreme case that Schubert outlines above). Better I think is the suggestion above where we do not move recently balanced regions. That should work for schubert's case.
          Hide
          Lars Hofhansl added a comment -

          Closing as dup of HBASE-3945, since that was suggested as the right approach to fix this.

          Show
          Lars Hofhansl added a comment - Closing as dup of HBASE-3945 , since that was suggested as the right approach to fix this.

            People

            • Assignee:
              Ted Yu
              Reporter:
              Ted Yu
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development