HBase
  1. HBase
  2. HBASE-5611

Replayed edits from regions that failed to open during recovery aren't removed from the global MemStore size

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.90.6
    • Fix Version/s: 0.92.2, 0.94.0, 0.95.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      This bug is rather easy to get if the TimeoutMonitor is on, else I think it's still possible to hit it if a region fails to open for more obscure reasons like HDFS errors.

      Consider a region that just went through distributed splitting and that's now being opened by a new RS. The first thing it does is to read the recovery files and put the edits in the MemStores. If this process takes a long time, the master will move that region away. At that point the edits are still accounted for in the global MemStore size but they are dropped when the HRegion gets cleaned up. It's completely invisible until the MemStoreFlusher needs to force flush a region and that none of them have edits:

      2012-03-21 00:33:39,303 DEBUG org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Flush thread woke up because memory above low water=5.9g
      2012-03-21 00:33:39,303 ERROR org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Cache flusher failed for entry null
      java.lang.IllegalStateException
              at com.google.common.base.Preconditions.checkState(Preconditions.java:129)
              at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:199)
              at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:223)
              at java.lang.Thread.run(Thread.java:662)
      

      The null here is a region. In my case I had so many edits in the MemStore during recovery that I'm over the low barrier although in fact I'm at 0. It happened yesterday and it still printing this out.

      To fix this we need to be able to decrease the global MemStore size when the region can't open.

      1. 5611-94.addendum
        3 kB
        Ted Yu
      2. 5611-94-v2.txt
        7 kB
        Ted Yu
      3. HBASE-5611-92.patch
        7 kB
        Jieshan Bean
      4. HBASE-5611-94-minorchange.patch
        7 kB
        Jieshan Bean
      5. HBASE-5611-trunk-v2-minorchange.patch
        7 kB
        Jieshan Bean

        Activity

        Jean-Daniel Cryans created issue -
        Jieshan Bean made changes -
        Field Original Value New Value
        Assignee Jieshan Bean [ jeason ]
        Jieshan Bean made changes -
        Attachment HBASE-5611-trunk.patch [ 12524294 ]
        Lars Hofhansl made changes -
        Fix Version/s 0.94.0 [ 12316419 ]
        Fix Version/s 0.94.1 [ 12320257 ]
        Lars Hofhansl made changes -
        Fix Version/s 0.94.1 [ 12320257 ]
        Fix Version/s 0.94.0 [ 12316419 ]
        Jieshan Bean made changes -
        Attachment HBASE-5611-trunk-v2.patch [ 12524411 ]
        Attachment HBASE-5611-94.patch [ 12524412 ]
        Jieshan Bean made changes -
        Attachment HBASE-5611-trunk-v2.patch [ 12524411 ]
        Jieshan Bean made changes -
        Attachment HBASE-5611-94.patch [ 12524412 ]
        Jieshan Bean made changes -
        Attachment HBASE-5611-94.patch [ 12524415 ]
        Attachment HBASE-5611-trunk-v2.patch [ 12524416 ]
        Ted Yu made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hadoop Flags Reviewed [ 10343 ]
        Jieshan Bean made changes -
        Attachment HBASE-5611-94.patch [ 12524415 ]
        Jieshan Bean made changes -
        Attachment HBASE-5611-94.patch [ 12524808 ]
        Ted Yu made changes -
        Comment [ -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12524808/HBASE-5611-94.patch
          against trunk revision .

            +1 @author. The patch does not contain any @author tags.

            -1 tests included. The patch doesn't appear to include any new or modified tests.
                                Please justify why no new tests are needed for this patch.
                                Also please list what manual steps were performed to verify this patch.

            -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1663//console

        This message is automatically generated. ]
        Jieshan Bean made changes -
        Attachment HBASE-5611-trunk-v2.patch [ 12524416 ]
        Jieshan Bean made changes -
        Attachment HBASE-5611-trunk-v2.patch [ 12524809 ]
        Jieshan Bean made changes -
        Attachment HBASE-5611-trunk-v2-minorchange.patch [ 12524810 ]
        Jieshan Bean made changes -
        Attachment HBASE-5611-92.patch [ 12524811 ]
        Attachment HBASE-5611-94-minorchange.patch [ 12524812 ]
        Jieshan Bean made changes -
        Attachment HBASE-5611-94.patch [ 12524808 ]
        Jieshan Bean made changes -
        Attachment HBASE-5611-trunk-v2.patch [ 12524809 ]
        Jieshan Bean made changes -
        Attachment HBASE-5611-trunk.patch [ 12524294 ]
        Lars Hofhansl made changes -
        Fix Version/s 0.94.0 [ 12316419 ]
        Fix Version/s 0.94.1 [ 12320257 ]
        Lars Hofhansl made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Ted Yu made changes -
        Attachment 5611-94.addendum [ 12524982 ]
        Ted Yu made changes -
        Resolution Fixed [ 1 ]
        Status Resolved [ 5 ] Reopened [ 4 ]
        Ted Yu made changes -
        Attachment 5611-94-v2.txt [ 12525090 ]
        Lars Hofhansl made changes -
        Status Reopened [ 4 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Lars Hofhansl made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        stack made changes -
        Fix Version/s 0.95.0 [ 12324094 ]
        Fix Version/s 0.94.0 [ 12316419 ]
        Fix Version/s 0.90.7 [ 12319481 ]
        Fix Version/s 0.92.2 [ 12319888 ]
        Fix Version/s 0.96.0 [ 12320040 ]
        Lars Hofhansl made changes -
        Fix Version/s 0.94.0 [ 12316419 ]
        Lars Hofhansl made changes -
        Fix Version/s 0.94.1 [ 12320257 ]
        Fix Version/s 0.94.0 [ 12316419 ]
        Lars Hofhansl made changes -
        Fix Version/s 0.94.0 [ 12316419 ]
        Fix Version/s 0.94.1 [ 12320257 ]
        stack made changes -
        Fix Version/s 0.92.2 [ 12319888 ]
        Fix Version/s 0.95.0 [ 12324094 ]
        stack made changes -
        Fix Version/s 0.95.0 [ 12324094 ]

          People

          • Assignee:
            Jieshan Bean
            Reporter:
            Jean-Daniel Cryans
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development