HBase
  1. HBase
  2. HBASE-3943

Propagate compaction status from HRegion to HServerLoad.RegionLoad

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: master, regionserver
    • Labels:
      None

      Description

      Load balancer works with HRegionInfo. However, compaction status is contained in HRegion.WriteState

      Users reported the following (Schubert Zhang):

      1. A region have many files, the compacting takes long time.
      2. But the balancer (default 5 minutes) close and move this region to
      another server.
      3. Then, the compacting start again.
      4. Then, then balancer close and move it to another server.

      Thus, the compacting cannot complete.
      Now, we set the balancer interval to 30 minutes to remission this issue.

      We need to propagate compaction status to HServerLoad.RegionLoad so that load balancer doesn't move the region which is being compacted.

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Resolved Resolved
          455d 5h 23m 1 Lars Hofhansl 29/Aug/12 22:54
          Lars Hofhansl made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Duplicate [ 3 ]
          Hide
          Lars Hofhansl added a comment -

          Closing as dup of HBASE-3945, since that was suggested as the right approach to fix this.

          Show
          Lars Hofhansl added a comment - Closing as dup of HBASE-3945 , since that was suggested as the right approach to fix this.
          Hide
          stack added a comment -

          While passing general outline of compaction status on regionserver might make sense giving a feel for what a cluster is up to, I do not think that it should be used as a factor balancing. For one the info the regionserver passes will likely be stale by the time the master goes to act on it (excepting the extreme case that Schubert outlines above). Better I think is the suggestion above where we do not move recently balanced regions. That should work for schubert's case.

          Show
          stack added a comment - While passing general outline of compaction status on regionserver might make sense giving a feel for what a cluster is up to, I do not think that it should be used as a factor balancing. For one the info the regionserver passes will likely be stale by the time the master goes to act on it (excepting the extreme case that Schubert outlines above). Better I think is the suggestion above where we do not move recently balanced regions. That should work for schubert's case.
          Hide
          Ted Yu added a comment -

          From Schubert:

          In my use case, the region-store size is aggresively set to be very large (such as 1GB, even 10GB), then, the time of compaction would take tens of minutes (e.g. 30 minutes).
           
          hbase.regionserver.msginterval=3000, it is the default value.
          

          The benefit of passing compaction status through RegionLoad is that load balancer wouldn't interrupt a half hour compaction 5 minutes after it starts (default value of hbase.balancer.period, some users make it longer to reduce impact of region reassignment).
          There is still a chance that compaction is interrupted 3 seconds after it starts, but the cost is much lower than the above scenario.

          Show
          Ted Yu added a comment - From Schubert: In my use case, the region-store size is aggresively set to be very large (such as 1GB, even 10GB), then, the time of compaction would take tens of minutes (e.g. 30 minutes). hbase.regionserver.msginterval=3000, it is the default value. The benefit of passing compaction status through RegionLoad is that load balancer wouldn't interrupt a half hour compaction 5 minutes after it starts (default value of hbase.balancer.period, some users make it longer to reduce impact of region reassignment). There is still a chance that compaction is interrupted 3 seconds after it starts, but the cost is much lower than the above scenario.
          Ted Yu made changes -
          Summary Propagate compaction status from HRegion to HRegionInfo Propagate compaction status from HRegion to HServerLoad.RegionLoad
          Description Load balancer works with HRegionInfo. However, compaction status is contained in HRegion.WriteState

          Users reported the following (Schubert Zhang):

          1. A region have many files, the compacting takes long time.
          2. But the balancer (default 5 minutes) close and move this region to
          another server.
          3. Then, the compacting start again.
          4. Then, then balancer close and move it to another server.

          Thus, the compacting cannot complete.
          Now, we set the balancer interval to 30 minutes to remission this issue.

          We need to propagate compaction status to HRegionInfo so that load balancer doesn't move the region being compacted.
          Load balancer works with HRegionInfo. However, compaction status is contained in HRegion.WriteState

          Users reported the following (Schubert Zhang):

          1. A region have many files, the compacting takes long time.
          2. But the balancer (default 5 minutes) close and move this region to
          another server.
          3. Then, the compacting start again.
          4. Then, then balancer close and move it to another server.

          Thus, the compacting cannot complete.
          Now, we set the balancer interval to 30 minutes to remission this issue.

          We need to propagate compaction status to HServerLoad.RegionLoad so that load balancer doesn't move the region which is being compacted.
          Hide
          Ted Yu added a comment -

          Thanks for the reminder, Stack.
          HRI isn't a proper carrier for compaction status.
          HRegionServer.buildServerLoad() has a chance to pass compaction status to Master. But in trunk, load balancer doesn't have access to HServerLoad (including RegionLoad).

          I agree that if a region was reassigned in round N, it shouldn't be reassigned again in round N+1, ideally.
          I will log another issue for that.

          Show
          Ted Yu added a comment - Thanks for the reminder, Stack. HRI isn't a proper carrier for compaction status. HRegionServer.buildServerLoad() has a chance to pass compaction status to Master. But in trunk, load balancer doesn't have access to HServerLoad (including RegionLoad). I agree that if a region was reassigned in round N, it shouldn't be reassigned again in round N+1, ideally. I will log another issue for that.
          Ted Yu made changes -
          Attachment 3943.txt [ 12481117 ]
          Hide
          stack added a comment -

          Are we mixing region schema and region metrics when we add this to HRI Ted? I'm not a fan of it. And do you think this the way to go? By the time the balancer goes to act on the info, it'll be stale, don't you think? It seems like the balancer issue that the boys were complaining off could be addressed by having the balancer not move regions it just moved in the previous balance run? Thanks Ted.

          Show
          stack added a comment - Are we mixing region schema and region metrics when we add this to HRI Ted? I'm not a fan of it. And do you think this the way to go? By the time the balancer goes to act on the info, it'll be stale, don't you think? It seems like the balancer issue that the boys were complaining off could be addressed by having the balancer not move regions it just moved in the previous balance run? Thanks Ted.
          Ted Yu made changes -
          Attachment 3943.txt [ 12481117 ]
          Hide
          Ted Yu added a comment -

          First attempt at solving this issue.

          Show
          Ted Yu added a comment - First attempt at solving this issue.
          Ted Yu made changes -
          Field Original Value New Value
          Link This issue is related to HBASE-3679 [ HBASE-3679 ]
          Ted Yu created issue -

            People

            • Assignee:
              Ted Yu
              Reporter:
              Ted Yu
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development