[HBASE-10148] [VisibilityController] Tolerate regions in recovery

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.98.0
• Fix Version/s: 0.98.0, 0.99.0
• Hadoop Flags: Reviewed
• Assignee: Anoop Sam John
• Reporter: Andrew Kyle Purtell
• Created: 12/Dec/13 23:35
• Updated: 21/Feb/15 23:30
• Resolved: 18/Dec/13 16:56

Description

Ted Yu reports that enabling distributed log replay by default, like:

Index: hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
===================================================================
--- hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java	(revision 1550575)
+++ hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java	(working copy)
@@ -794,7 +794,7 @@

   /** Conf key that enables unflushed WAL edits directly being replayed to region servers */
   public static final String DISTRIBUTED_LOG_REPLAY_KEY = "hbase.master.distributed.log.replay";
-  public static final boolean DEFAULT_DISTRIBUTED_LOG_REPLAY_CONFIG = false;
+  public static final boolean DEFAULT_DISTRIBUTED_LOG_REPLAY_CONFIG = true;
   public static final String DISALLOW_WRITES_IN_RECOVERING =
       "hbase.regionserver.disallow.writes.when.recovering";
   public static final boolean DEFAULT_DISALLOW_WRITES_IN_RECOVERING_CONFIG = false;

causes TestVisibilityController#testAddVisibilityLabelsOnRSRestart to fail. It reveals an issue with label operations if the label table is recovering:

2013-12-12 14:53:53,133 DEBUG [RpcServer.handler=2,port=58108] visibility.VisibilityController(1046): Adding the label XYZ2013-12-12 14:53:53,137 ERROR [RpcServer.handler=2,port=58108] visibility.VisibilityController(1074): org.apache.hadoop.hbase.exceptions.RegionInRecoveryException: hbase:labels,,1386888826648.f14a399ba85cbb42c2c3b7547bf17c65. is recovering
2013-12-12 14:53:53,151 DEBUG [main] visibility.TestVisibilityLabels(405): response from addLabels: result {
  exception {
    name: "org.apache.hadoop.hbase.exceptions.RegionInRecoveryException"
    value: "org.apache.hadoop.hbase.exceptions.RegionInRecoveryException: hbase:labels,,1386888826648.f14a399ba85cbb42c2c3b7547bf17c65. is recovering at org.apache.hadoop.hbase.regionserver.HRegion.startRegionOperation(HRegion.java:5555) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1763) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1749) at org.apache.hadoop.hbase.security.visibility.VisibilityController.getExistingLabelsWithAuths(VisibilityController.java:1096) at org.apache.hadoop.hbase.security.visibility.VisibilityController.postBatchMutate(VisibilityController.java:672)"

Should we try to ride over this?

Issue Links

• depends upon: HBASE-10155 HRegion isRecovering state is wrongly coming in postOpen hook
• is blocked by: HBASE-10157 Provide CP hook post log replay
• relates to: HBASE-10161 [AccessController] Tolerate regions in recovery