HBase
  1. HBase
  2. HBASE-5611

Replayed edits from regions that failed to open during recovery aren't removed from the global MemStore size

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.90.6
    • Fix Version/s: 0.92.2, 0.94.0, 0.95.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      This bug is rather easy to get if the TimeoutMonitor is on, else I think it's still possible to hit it if a region fails to open for more obscure reasons like HDFS errors.

      Consider a region that just went through distributed splitting and that's now being opened by a new RS. The first thing it does is to read the recovery files and put the edits in the MemStores. If this process takes a long time, the master will move that region away. At that point the edits are still accounted for in the global MemStore size but they are dropped when the HRegion gets cleaned up. It's completely invisible until the MemStoreFlusher needs to force flush a region and that none of them have edits:

      2012-03-21 00:33:39,303 DEBUG org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Flush thread woke up because memory above low water=5.9g
      2012-03-21 00:33:39,303 ERROR org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Cache flusher failed for entry null
      java.lang.IllegalStateException
              at com.google.common.base.Preconditions.checkState(Preconditions.java:129)
              at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:199)
              at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:223)
              at java.lang.Thread.run(Thread.java:662)
      

      The null here is a region. In my case I had so many edits in the MemStore during recovery that I'm over the low barrier although in fact I'm at 0. It happened yesterday and it still printing this out.

      To fix this we need to be able to decrease the global MemStore size when the region can't open.

      1. 5611-94.addendum
        3 kB
        Ted Yu
      2. 5611-94-v2.txt
        7 kB
        Ted Yu
      3. HBASE-5611-92.patch
        7 kB
        Jieshan Bean
      4. HBASE-5611-94-minorchange.patch
        7 kB
        Jieshan Bean
      5. HBASE-5611-trunk-v2-minorchange.patch
        7 kB
        Jieshan Bean

        Activity

        No work has yet been logged on this issue.

          People

          • Assignee:
            Jieshan Bean
            Reporter:
            Jean-Daniel Cryans
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development