HRegionServer won't go down since we added in new LruBlockCache

• Key: HBASE-1602
• Type: Bug
• Status: Closed
• Resolution: Fixed
• Priority: Major
• Assignee: stack
• Reporter: stack
• Fix Version/s: 0.20.0

Description

New LruBlockCache uses thread excecutor scheduling stats dumping. The scheduled excecutor needs to be cancelled else it just stays running stopping the HRS going down.

Comments

[stack]

Add a shutdown the the BlockCache interface. Change SimpleBlockCache and LruBlockCache so they implement new shutdown.

[stack]

Committed after testing. Easy to reproduce the problem.

[stack]

Committing the below under the aegis if this issue:

Index: src/java/org/apache/hadoop/hbase/master/HMaster.java
===================================================================
--- src/java/org/apache/hadoop/hbase/master/HMaster.java        (revision 790761)
+++ src/java/org/apache/hadoop/hbase/master/HMaster.java        (working copy)
@@ -271,6 +271,12 @@
   private void bootstrap() throws IOException {
     LOG.info("BOOTSTRAP: creating ROOT and first META regions");
     try {
+      // Bootstrapping, make sure blockcache is off.  Else, one will be
+      // created here in bootstap and it'll need to be cleaned up.  Better to
+      // not make it in first place.  Turn off block caching for bootstrap.
+      // Enable after.
+      setBlockCaching(HRegionInfo.ROOT_REGIONINFO, false);
+      setBlockCaching(HRegionInfo.FIRST_META_REGIONINFO, false);
       HRegion root = HRegion.createHRegion(HRegionInfo.ROOT_REGIONINFO,
         this.rootdir, this.conf);
       HRegion meta = HRegion.createHRegion(HRegionInfo.FIRST_META_REGIONINFO,
@@ -281,6 +287,8 @@
       root.getLog().closeAndDelete();
       meta.close();
       meta.getLog().closeAndDelete();
+      setBlockCaching(HRegionInfo.ROOT_REGIONINFO, true);
+      setBlockCaching(HRegionInfo.FIRST_META_REGIONINFO, true);
     } catch (IOException e) {
       e = RemoteExceptionHandler.checkIOException(e);
       LOG.error("bootstrap", e);
@@ -288,6 +296,16 @@
     }
   }
 
+  /*
+   * @param hri Set all family block caching to <code>b</code>
+   * @param b
+   */
+  private void setBlockCaching(final HRegionInfo hri, final boolean b) {
+    for (HColumnDescriptor hcd: hri.getTableDesc().families.values()) {
+      hcd.setBlockCacheEnabled(b);
+    }
+  }
+

There was a subtle condition where a BlockCache would be made in the master if we did a bootstrap. If the block cache was the default LruBlockCache, then it needs a shutdown to go down else the JVM stays up.

[Andrew Purtell]

Ah, that explains that. Thanks stack.