[HBASE-1091] shell tools -> close_region does not work for regions that did not deploy properly on startup

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.19.0
• Component/s: master
• Assignee: Michael Stack
• Reporter: Andrew Kyle Purtell
• Created: 28/Dec/08 21:26
• Updated: 13/Sep/09 22:26
• Resolved: 30/Dec/08 07:03

Description

2008-12-28 16:14:32,292 INFO org.apache.hadoop.hbase.master.HMaster: Marking result_domain,,1230404458436 as closed on 10.30.94.48:60020; cleaning SERVER + STARTCODE; master will tell regionserver to close region on next heartbeat
2008-12-28 16:14:32,293 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 4 on 60000, call modifyTable([B@64c6e290, 4, [Lorg.apache.hadoop.io.Writable;@3d448fa0) from 10.30.94.51:55082: error: java.io.IOException: java.lang.IllegalStateException: Cannot transition to closing from any other state. Region: result_domain,,1230404458436
java.io.IOException: java.lang.IllegalStateException: Cannot transition to closing from any other state. Region: result_domain,,1230404458436
at org.apache.hadoop.hbase.master.RegionManager.setClosing(RegionManager.java:797)
at org.apache.hadoop.hbase.master.HMaster.modifyTable(HMaster.java:836)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)

Clearing 'info:server' and 'info:serverstartcode' for the affected region has no effect either.

The region in question did not properly deploy:

2008-12-28 15:58:44,857 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_CLOSE: result_domain,,1230404458436: java.io.IOException: Could not obtain block: blk_-1589286342406311350_4245713 file=/data/hbase/result_domain/43721464/ErrorResult/mapfiles/284087383342325737/index
at org.apache.hadoop.dfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1472)
at org.apache.hadoop.dfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1322)
at org.apache.hadoop.dfs.DFSClient$DFSInputStream.read(DFSClient.java:1427)
at java.io.DataInputStream.readFully(DataInputStream.java:178)
at java.io.DataInputStream.readFully(DataInputStream.java:152)
at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1453)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1431)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1420)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1415)
at org.apache.hadoop.io.MapFile$Reader.open(MapFile.java:292)
at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.<init>(HBaseMapFile.java:81)
at org.apache.hadoop.hbase.io.BloomFilterMapFile$Reader.<init>(BloomFilterMapFile.java:66)
at org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:443)
at org.apache.hadoop.hbase.regionserver.HStore.setupReaders(HStore.java:254)
at org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:239)
at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:1624)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:270)
at org.apache.hadoop.hbase.regionserver.HRegionServer.instantiateRegion(HRegionServer.java:1360)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:1331)
at org.apache.hadoop.hbase.regionserver.HRegionServer$Worker.run(HRegionServer.java:1247)
at java.lang.Thread.run(Thread.java:619)
from 10.30.94.34:60020

I am trying to force a redeployment in an attempt to manually recover the region.