[HBASE-1104] Doubly-assigned regions redux

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.19.0

Environment

pset cluster with TRUNK.

• Assignee: Jim Kellerman
• Reporter: Michael Stack
• Created: 31/Dec/08 23:07
• Updated: 13/Sep/09 22:26
• Resolved: 09/Jan/09 01:50

Description

Testing, I see doubly assigned regions. Below is from master log for TestTable,0000135598,1230761605500.

2008-12-31 22:13:35,528 [IPC Server handler 2 on 60000] INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_SPLIT: TestTable,0000116170,1230761152219: TestTable,0000116170,1230761152219 split; daughters: TestTable,0000116170,1230761605500, TestTable,0000135598,1230761605500 from XX.XX.XX.142:60020
2008-12-31 22:13:35,528 [IPC Server handler 2 on 60000] INFO org.apache.hadoop.hbase.master.RegionManager: assigning region TestTable,0000135598,1230761605500 to server XX.XX.XX.142:60020
2008-12-31 22:13:38,561 [IPC Server handler 6 on 60000] INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: TestTable,0000135598,1230761605500 from XX.XX.XX.142:60020
2008-12-31 22:13:38,562 [HMaster] INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: TestTable,0000135598,1230761605500 open on XX.XX.XX.142:60020
2008-12-31 22:13:38,562 [HMaster] INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row TestTable,0000135598,1230761605500 in region .META.,,1 with startcode 1230759988953 and server XX.XX.XX.142:60020
2008-12-31 22:13:44,640 [IPC Server handler 4 on 60000] DEBUG org.apache.hadoop.hbase.master.RegionManager: Going to close region TestTable,0000135598,1230761605500
2008-12-31 22:13:50,441 [IPC Server handler 9 on 60000] INFO org.apache.hadoop.hbase.master.RegionManager: assigning region TestTable,0000135598,1230761605500 to server XX.XX.XX.139:60020
2008-12-31 22:13:53,457 [IPC Server handler 5 on 60000] INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN: TestTable,0000135598,1230761605500 from XX.XX.XX.139:60020
2008-12-31 22:13:53,458 [IPC Server handler 5 on 60000] INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: TestTable,0000135598,1230761605500 from XX.XX.XX.139:60020
2008-12-31 22:13:53,458 [HMaster] INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: TestTable,0000135598,1230761605500 open on XX.XX.XX.139:60020
2008-12-31 22:13:53,458 [HMaster] INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row TestTable,0000135598,1230761605500 in region .META.,,1 with startcode 1230759988788 and server XX.XX.XX.139:60020
2008-12-31 22:13:53,688 [IPC Server handler 6 on 60000] INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_CLOSE: TestTable,0000135598,1230761605500 from XX.XX.XX.142:60020
2008-12-31 22:13:53,688 [HMaster] DEBUG org.apache.hadoop.hbase.master.HMaster: Processing todo: ProcessRegionClose of TestTable,0000135598,1230761605500, false
2008-12-31 22:13:54,263 [IPC Server handler 7 on 60000] INFO org.apache.hadoop.hbase.master.RegionManager: assigning region TestTable,0000135598,1230761605500 to server XX.XX.XX.141:60020
2008-12-31 22:13:57,273 [IPC Server handler 9 on 60000] INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN: TestTable,0000135598,1230761605500 from XX.XX.XX.141:60020
2008-12-31 22:14:03,917 [IPC Server handler 0 on 60000] INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: TestTable,0000135598,1230761605500 from XX.XX.XX.141:60020
2008-12-31 22:14:03,917 [HMaster] INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: TestTable,0000135598,1230761605500 open on XX.XX.XX.141:60020
2008-12-31 22:14:03,918 [HMaster] INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row TestTable,0000135598,1230761605500 in region .META.,,1 with startcode 1230759989031 and server XX.XX.XX.141:60020
2008-12-31 22:14:29,350 [RegionManager.metaScanner] DEBUG org.apache.hadoop.hbase.master.BaseScanner: TestTable,0000135598,1230761605500 no longer has references to TestTable,0000116170,1230761152219

See how we choose to assign before we get the close back from the regionserver.

Sub-Tasks

• IllegalStateException: Cannot set a region to be closed it it was not already marked as closing: Closed
• Does not recover if HRS carrying -ROOT- goes down: Closed