[HBASE-21425] 2.1.1 fails to start over 1.x data; namespace not assigned

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 3.0.0-alpha-1, 2.2.0, 2.0.3, 2.1.2
• Component/s: amv2
• Hadoop Flags: Reviewed
• Assignee: Michael Stack
• Reporter: Michael Stack
• Created: 01/Nov/18 20:01
• Updated: 23/Apr/19 11:51
• Resolved: 03/Nov/18 16:47

Description

I tested hbase-2.1.1 starting up over data written by branch-1.4. It failed because the TableStateManager, as part of its startup, failed its migration of table state from zookeeper to hbase:meta table. This is exception:

2018-11-01 10:49:33,678 ERROR [master/kalashnikov:16000:becomeActiveMaster] master.TableStateManager: Unable to get table hbase:namespace state
org.apache.hadoop.hbase.master.TableStateManager$TableStateNotFoundException: hbase:namespace
  at org.apache.hadoop.hbase.master.TableStateManager.getTableState(TableStateManager.java:215)
  at org.apache.hadoop.hbase.master.TableStateManager.isTableState(TableStateManager.java:147)
  at org.apache.hadoop.hbase.master.assignment.AssignmentManager.isTableEnabled(AssignmentManager.java:327)
  at org.apache.hadoop.hbase.master.assignment.AssignmentManager.lambda$processOfflineRegions$3(AssignmentManager.java:1236)
  at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174)
  at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
  at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
  at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
  at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
  at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
  at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
  at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
  at org.apache.hadoop.hbase.master.assignment.AssignmentManager.processOfflineRegions(AssignmentManager.java:1237)
  at org.apache.hadoop.hbase.master.assignment.AssignmentManager.joinCluster(AssignmentManager.java:1218)
  at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:1001)
  at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2257)
  at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:583)
  at java.lang.Thread.run(Thread.java:748)

This happens inside in processOfflineRegions so result of above exception is that procedures are not scheduled; i.e. namespace table assign for one is not assigned.

Issue Links

• is related to: HBASE-21035 Meta Table should be able to online even if all procedures are lost
• links to: Review Board (branch-2.1)