HBase
  1. HBase
  2. HBASE-138

[hbase] Under load, regions become extremely large and eventually cause region servers to become unresponsive

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      When attempting to write to HBase as fast as possible, HBase accepts puts at a reasonably high rate for a while, and then the rate begins to drop off, ultimately culminating in exceptions reaching client code. In my testing, I was able to write about 370 10KB records a second to HBase until I reach around 1 million rows written. At that point, a moderate to large number of exceptions - NotServingRegionException, WrongRegionException, region offline, etc - begin reaching the client code. This appears to be because the retry-and-wait logic in HTable runs out of retries and fails.

      Looking at mapfiles for the regions from the command line shows that some of the mapfiles are between 1 and 2 GB in size, much more than the stated file size limit. Talking with Stack, one possible explanation for this is that the RegionServer is not choosing to compact files often enough, leading to many small mapfiles, which in turn leads to a few overlarge mapfiles. Then, when the time comes to do a split or "major" compaction, it takes an unexpectedly long time to complete these operations. This translates into errors for the client application.

      If I back off the import process and give the cluster some quiet time, some splits and compactions clearly do take place, because the number of regions go up and the number of mapfiles/region goes down. I can then begin writing again in earnest for a short period of time until the problem begins again.

      Both Marc Harris and myself have seen this behavior.

      1. split-v12.patch
        25 kB
        stack
      2. split-v11.patch
        24 kB
        stack
      3. split-v10.patch
        18 kB
        stack
      4. split-v9.patch
        17 kB
        stack
      5. split-v8.patch
        17 kB
        stack
      6. split.patch
        12 kB
        stack

        Issue Links

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              Bryan Duxbury
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development