[HBASE-1098] IllegalStateException: Cannot set a region to be closed it it was not already marked as closing

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.19.0

Environment

apurtel cluster

• Assignee: Jim Kellerman
• Reporter: Michael Stack
• Created: 29/Dec/08 23:53
• Updated: 13/Sep/09 22:26
• Resolved: 09/Jan/09 01:49

Description

content,ec0ec4d005ee7da14cbd1182339f8cf4,1230416299322 is problem region. below is snippet from master log. Looks to me like things are proceeding in order.

2008-12-29 12:19:32,074 DEBUG org.apache.hadoop.hbase.master.RegionManager: Server XX.XX.XX.51:60020 is overloaded. Server load: 74 avg: 48.0, slop: 0.1
2008-12-29 12:19:32,074 DEBUG org.apache.hadoop.hbase.master.RegionManager: Choosing to reassign 26 regions. mostLoadedRegions has 10 regions in it.
2008-12-29 12:19:32,074 DEBUG org.apache.hadoop.hbase.master.RegionManager: Going to close region content,3ea6e20cf48a32fc063912ccae0e960e,1230416307426
2008-12-29 12:19:32,074 DEBUG org.apache.hadoop.hbase.master.RegionManager: Going to close region content,07730cecbb5b13364cbe17af5778f983,1230448573272
2008-12-29 12:19:32,074 DEBUG org.apache.hadoop.hbase.master.RegionManager: Going to close region content,e8c750a5983137306e6b3a84eb1fd6e6,1230424193882
2008-12-29 12:19:32,074 DEBUG org.apache.hadoop.hbase.master.RegionManager: Going to close region content,29be0a07dece4e9bb318c4d253807287,1230356680860
2008-12-29 12:19:32,074 DEBUG org.apache.hadoop.hbase.master.RegionManager: Going to close region content,23893c82d792ff5c9d3ff99123bc1773,1230390013767
2008-12-29 12:19:32,074 DEBUG org.apache.hadoop.hbase.master.RegionManager: Going to close region content,e76475012b59500d25000aa7658bc279,1230498762282
2008-12-29 12:19:32,074 DEBUG org.apache.hadoop.hbase.master.RegionManager: Going to close region result_domain,cn.com.sg.www,1230498785950
2008-12-29 12:19:32,074 DEBUG org.apache.hadoop.hbase.master.RegionManager: Going to close region content,c7b7e7a475765e16d0bbf1e51c95e85d,1230519854914
2008-12-29 12:19:32,074 DEBUG org.apache.hadoop.hbase.master.RegionManager: Going to close region content,ec0ec4d005ee7da14cbd1182339f8cf4,1230416299322
2008-12-29 12:19:32,074 DEBUG org.apache.hadoop.hbase.master.RegionManager: Going to close region content,e16b2ea9f6da8b32fa96ab3490e3d849,1230424086678
2008-12-29 12:19:32,074 INFO org.apache.hadoop.hbase.master.RegionManager: Skipped 0 region(s) that are in transition states
2008-12-29 12:19:32,076 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: PendingOpenOperation from XX.XX.XX.51:60020
2008-12-29 12:19:32,076 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: content,42b19221f3a60945595b18017d6bacfa,1230357228381 open on XX.XX.XX.51:60020
2008-12-29 12:19:32,076 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row content,42b19221f3a60945595b18017d6bacfa,1230357228381 in region .META.,,1 with startcode 1230571104777 and server XX.XX.XX.51:60020
2008-12-29 12:19:32,077 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: PendingOpenOperation from XX.XX.XX.51:60020
2008-12-29 12:19:32,077 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: content,3c619bc7a2c824fb3c4f5d747b0f2f12,1230354673241 open on XX.XX.XX.51:60020
2008-12-29 12:19:32,077 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row content,3c619bc7a2c824fb3c4f5d747b0f2f12,1230354673241 in region .META.,,1 with startcode 1230571104777 and server XX.XX.XX.51:60020
2008-12-29 12:19:32,078 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: ProcessRegionClose of content,36946541ed9a62f419cf7238d32a6a38,1230448587552, false
2008-12-29 12:19:32,078 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: ProcessRegionClose of content,58221f5f3a79b7dd5e23c339c507a7c6,1230487059978, false
2008-12-29 12:19:32,078 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: ProcessRegionClose of content,94b170cae87992c33c3913d74bb257e4,1230498360162, false
2008-12-29 12:19:32,078 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: ProcessRegionClose of content,bb34d860d76c1eabbe2eb7d2ce8148d9,1230416266955, false
2008-12-29 12:19:32,078 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: ProcessRegionClose of content,8ac5fbdd63b313ce252838a189c73438,1230530000480, false
2008-12-29 12:19:32,078 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: ProcessRegionClose of content,f9f5b2d035349cf4292ed7dd7e9823f0,1230505507313, false
2008-12-29 12:19:32,078 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: ProcessRegionClose of content,4b30e2a8b7949ed62e097824969f8bca,1230448501152, false
2008-12-29 12:19:32,078 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: ProcessRegionClose of content,0c4b7c2825f674510a511233e9e0c539,1230530122456, false
2008-12-29 12:19:32,078 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: ProcessRegionClose of content,43f83c42e08bb0f4b1f6d1125f39f018,1230498357831, false
2008-12-29 12:19:32,299 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region content,43f83c42e08bb0f4b1f6d1125f39f018,1230498357831 to server XX.XX.XX.34:60020
2008-12-29 12:19:32,300 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region content,bb34d860d76c1eabbe2eb7d2ce8148d9,1230416266955 to server XX.XX.XX.34:60020
2008-12-29 12:19:32,301 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region content,94b170cae87992c33c3913d74bb257e4,1230498360162 to server XX.XX.XX.34:60020
2008-12-29 12:19:32,301 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region content,8ac5fbdd63b313ce252838a189c73438,1230530000480 to server XX.XX.XX.34:60020
2008-12-29 12:19:32,302 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region content,36946541ed9a62f419cf7238d32a6a38,1230448587552 to server XX.XX.XX.34:60020
2008-12-29 12:19:32,302 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region content,58221f5f3a79b7dd5e23c339c507a7c6,1230487059978 to server XX.XX.XX.34:60020
2008-12-29 12:19:32,303 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region content,f9f5b2d035349cf4292ed7dd7e9823f0,1230505507313 to server XX.XX.XX.34:60020
2008-12-29 12:19:32,303 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region content,4b30e2a8b7949ed62e097824969f8bca,1230448501152 to server XX.XX.XX.34:60020
2008-12-29 12:19:32,304 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region content,0c4b7c2825f674510a511233e9e0c539,1230530122456 to server XX.XX.XX.34:60020
2008-12-29 12:19:35,080 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_CLOSE: content,3ea6e20cf48a32fc063912ccae0e960e,1230416307426 from XX.XX.XX.51:60020
2008-12-29 12:19:35,081 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_CLOSE: content,07730cecbb5b13364cbe17af5778f983,1230448573272 from XX.XX.XX.51:60020
2008-12-29 12:19:35,081 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: ProcessRegionClose of content,3ea6e20cf48a32fc063912ccae0e960e,1230416307426, false
2008-12-29 12:19:35,081 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_CLOSE: content,e8c750a5983137306e6b3a84eb1fd6e6,1230424193882 from XX.XX.XX.51:60020
2008-12-29 12:19:35,081 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: ProcessRegionClose of content,07730cecbb5b13364cbe17af5778f983,1230448573272, false
2008-12-29 12:19:35,081 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_CLOSE: content,29be0a07dece4e9bb318c4d253807287,1230356680860 from XX.XX.XX.51:60020
2008-12-29 12:19:35,081 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: ProcessRegionClose of content,e8c750a5983137306e6b3a84eb1fd6e6,1230424193882, false
2008-12-29 12:19:35,081 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_CLOSE: content,23893c82d792ff5c9d3ff99123bc1773,1230390013767 from XX.XX.XX.51:60020
2008-12-29 12:19:35,081 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: ProcessRegionClose of content,29be0a07dece4e9bb318c4d253807287,1230356680860, false
2008-12-29 12:19:35,081 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_CLOSE: content,e76475012b59500d25000aa7658bc279,1230498762282 from XX.XX.XX.51:60020
2008-12-29 12:19:35,081 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: ProcessRegionClose of content,23893c82d792ff5c9d3ff99123bc1773,1230390013767, false
2008-12-29 12:19:35,081 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_CLOSE: result_domain,cn.com.sg.www,1230498785950 from XX.XX.XX.51:60020
2008-12-29 12:19:35,081 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: ProcessRegionClose of content,e76475012b59500d25000aa7658bc279,1230498762282, false
2008-12-29 12:19:35,081 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_CLOSE: content,c7b7e7a475765e16d0bbf1e51c95e85d,1230519854914 from XX.XX.XX.51:60020
2008-12-29 12:19:35,081 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: ProcessRegionClose of result_domain,cn.com.sg.www,1230498785950, false
2008-12-29 12:19:35,081 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_CLOSE: content,ec0ec4d005ee7da14cbd1182339f8cf4,1230416299322 from XX.XX.XX.51:60020
2008-12-29 12:19:35,081 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: ProcessRegionClose of content,c7b7e7a475765e16d0bbf1e51c95e85d,1230519854914, false
2008-12-29 12:19:35,081 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_CLOSE: content,e16b2ea9f6da8b32fa96ab3490e3d849,1230424086678 from XX.XX.XX.51:60020
2008-12-29 12:19:35,081 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: ProcessRegionClose of content,ec0ec4d005ee7da14cbd1182339f8cf4,1230416299322, false
2008-12-29 12:19:35,082 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: ProcessRegionClose of content,e16b2ea9f6da8b32fa96ab3490e3d849,1230424086678, false
2008-12-29 12:19:35,083 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 9 on 60000, call regionServerReport(address: XX.XX.XX.51:60020, startcode: 1230571104777, load: (requests=0, regions=64, usedHeap=99, maxHeap=1777), [Lorg.
apache.hadoop.hbase.HMsg;@3e2dce4e, [Lorg.apache.hadoop.hbase.HRegionInfo;@314382c6) from XX.XX.XX.51:55371: error: java.io.IOException: java.lang.IllegalStateException: Cannot set a region to be closed if it was not already marked as clo
sing. Region: content,ec0ec4d005ee7da14cbd1182339f8cf4,1230416299322
java.io.IOException: java.lang.IllegalStateException: Cannot set a region to be closed if it was not already marked as closing. Region: content,ec0ec4d005ee7da14cbd1182339f8cf4,1230416299322
        at org.apache.hadoop.hbase.master.RegionManager$RegionState.setClosed(RegionManager.java:1108)
        at org.apache.hadoop.hbase.master.RegionManager.setClosed(RegionManager.java:816)
        at org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:377)
        at org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:327)
        at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:240)
        at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:570)
        at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)