[HBASE-7161] Table does not come out of 'enabling' state

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Later
• Affects Version/s: 0.94.2
• Assignee: Unassigned
• Reporter: Devaraj Das
• Created: 14/Nov/12 22:20
• Updated: 15/Jun/22 00:19
• Resolved: 01/Jun/20 19:03

Description

I was running a test, and the test failed because a table didn't get 'enabled' in the timeframe the test expected. When I checked the state of the table on ZK, it showed the state as 'enabling'.

When I dug up the master logs, found that the BulkAssigner.bulkAssign returned false (the first line in the logs below), and the table never became 'enabled'. There was one region which could not be opened in the time bulkAssign ran (and that got 'opened' after the bulkAssign method returned). Also the table could not be enabled later on (from the hbase shell, for example), since the state of the table on ZK was 'enabling' (as opposed to 'disabled' and the table-state checks would fail).

2012-11-13 06:41:27,257 INFO org.apache.hadoop.hbase.master.handler.EnableTableHandler: Enabled table is done=false 
2012-11-13 06:41:49,569 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_OPENING, server=hrt20n32.foo.net,60020,1352782575357, region=0c0f9c71a81112f07c8f0ea130a65d05
2012-11-13 06:41:49,579 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_OPENING, server=hrt20n32.foo.net,60020,1352782575357, region=0c0f9c71a81112f07c8f0ea130a65d05
2012-11-13 06:41:49,586 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_OPENED, server=hrt20n32.foo.net,60020,1352782575357, region=0c0f9c71a81112f07c8f0ea130a65d05
2012-11-13 06:41:49,586 DEBUG org.apache.hadoop.hbase.master.handler.OpenedRegionHandler: Handling OPENED event for loadtest_d1,44444444,1352788441221.0c0f9c71a81112f07c8f0ea130a65d05. from hrt20n32.foo.net,60020,1352782575357; deleting unassigned node
2012-11-13 06:41:49,586 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x13af81eec6f0004 Deleting existing unassigned node for 0c0f9c71a81112f07c8f0ea130a65d05 that is in expected state RS_ZK_REGION_OPENED
2012-11-13 06:41:49,589 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: The znode of region loadtest_d1,44444444,1352788441221.0c0f9c71a81112f07c8f0ea130a65d05. has been deleted.
2012-11-13 06:41:49,589 INFO org.apache.hadoop.hbase.master.AssignmentManager: The master has opened the region loadtest_d1,44444444,1352788441221.0c0f9c71a81112f07c8f0ea130a65d05. that was online on hrt20n32.foo.net,60020,1352782575357

The client (that invoked HBA.enableTable) gave up eventually (enableTable invokes isTableEnabled in a loop and in this case it was always returning false).

The handling on the master side for regions that take longer to get 'opened' can be improved.