Log: 101HM the region is assigning, ZK restart because of exception: 2011-07-21 07:05:44,712 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:20000-0x1314ac5addb003e Async create of unassigned node for ffe0f4ab26eaa1796d2f99565ff24012 with OFFLINE state 2011-07-21 07:05:46,661 DEBUG org.apache.hadoop.hbase.master.AssignmentManager$CreateUnassignedAsyncCallback: rs=test1,282139,1311216321932.ffe0f4ab26eaa1796d2f99565ff24012. state=OFFLINE, ts=1311231944712, server=158-1-91-105,20020,1311231874887 2011-07-21 07:05:46,837 DEBUG org.apache.hadoop.hbase.master.AssignmentManager$ExistsUnassignedAsyncCallback: rs=test1,282139,1311216321932.ffe0f4ab26eaa1796d2f99565ff24012. state=OFFLINE, ts=1311231944712 ... 2011-07-21 07:05:55,734 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x1314ac5addb003f, likely server has closed socket, closing socket connection and attempting reconnect 2011-07-21 07:05:55,835 FATAL org.apache.hadoop.hbase.master.HMaster: Error deleting OPENED node in ZK for transition ZK node (2c5d23581fc848c89e5be0555086cf30) org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/unassigned/2c5d23581fc848c89e5be0555086cf30 at org.apache.zookeeper.KeeperException.create(KeeperException.java:90) at org.apache.zookeeper.KeeperException.create(KeeperException.java:42) at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:738) at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:962) ... 2011-07-21 07:08:03,239 INFO org.apache.hadoop.hbase.master.metrics.MasterMetrics: Initialized 2011-07-21 07:08:03,288 INFO org.apache.hadoop.hbase.master.ActiveMasterManager: Another master is the active master, 158-1-91-103:20000; waiting to become the next active master 103HM get active,assign three times. Thu Jul 21 07:04:45 UTC 2011 Starting master on 158-1-91-103 ulimit -n 640000 2011-07-21 07:04:45,482 INFO org.apache.hadoop.hbase.ipc.HBaseRpcMetrics: Initializing RPC Metrics with hostName=HMaster, port=20000 ... 2011-07-21 07:15:13,719 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:20000-0x1314ac5addb0042-0x1314ac5addb0042 Creating (or updating) unassigned node for ffe0f4ab26eaa1796d2f99565ff24012 with OFFLINE state 2011-07-21 07:30:26,526 INFO org.apache.hadoop.hbase.master.AssignmentManager: Processing region test1,282139,1311216321932.ffe0f4ab26eaa1796d2f99565ff24012. in state M_ZK_REGION_OFFLINE 2011-07-21 07:30:26,562 DEBUG org.apache.hadoop.hbase.master.handler.ClosedRegionHandler: Handling CLOSED event for ffe0f4ab26eaa1796d2f99565ff24012 2011-07-21 07:30:26,562 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=test1,282139,1311216321932.ffe0f4ab26eaa1796d2f99565ff24012. state=OFFLINE, ts=1311232513720 2011-07-21 07:30:26,562 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:20000-0x1314ac5addb0042-0x1314ac5addb0042 Creating (or updating) unassigned node for ffe0f4ab26eaa1796d2f99565ff24012 with OFFLINE state 2011-07-21 07:30:26,577 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: No previous transition plan was found (or we are ignoring an existing plan) for test1,282139,1311216321932.ffe0f4ab26eaa1796d2f99565ff24012. so generated a random one; hri=test1,282139,1311216321932.ffe0f4ab26eaa1796d2f99565ff24012., src=, dest=158-1-91-105,20020,1311231874887; 3 (online=3, exclude=null) available servers 2011-07-21 07:30:26,577 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Assigning region test1,282139,1311216321932.ffe0f4ab26eaa1796d2f99565ff24012. to 158-1-91-105,20020,1311231874887 2011-07-21 07:30:26,603 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=M_ZK_REGION_OFFLINE, server=158-1-91-103:20000, region=ffe0f4ab26eaa1796d2f99565ff24012 2011-07-21 07:42:08,737 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: test1,282139,1311216321932.ffe0f4ab26eaa1796d2f99565ff24012. state=PENDING_OPEN, ts=1311233942949 2011-07-21 07:42:08,737 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been PENDING_OPEN for too long, reassigning region=test1,282139,1311216321932.ffe0f4ab26eaa1796d2f99565ff24012. 2011-07-21 07:42:49,043 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=test1,282139,1311216321932.ffe0f4ab26eaa1796d2f99565ff24012. state=PENDING_OPEN, ts=1311234169040 2011-07-21 07:42:49,043 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: No previous transition plan was found (or we are ignoring an existing plan) for test1,282139,1311216321932.ffe0f4ab26eaa1796d2f99565ff24012. so generated a random one; hri=test1,282139,1311216321932.ffe0f4ab26eaa1796d2f99565ff24012., src=, dest=158-1-91-105,20020,1311231874887; 3 (online=3, exclude=null) available servers 2011-07-21 07:42:49,043 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Assigning region test1,282139,1311216321932.ffe0f4ab26eaa1796d2f99565ff24012. to 158-1-91-105,20020,1311231874887 2011-07-21 07:46:25,809 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: test1,282139,1311216321932.ffe0f4ab26eaa1796d2f99565ff24012. state=PENDING_OPEN, ts=1311234201519 2011-07-21 07:46:25,809 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been PENDING_OPEN for too long, reassigning region=test1,282139,1311216321932.ffe0f4ab26eaa1796d2f99565ff24012. 2011-07-21 07:46:31,872 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=test1,282139,1311216321932.ffe0f4ab26eaa1796d2f99565ff24012. state=PENDING_OPEN, ts=1311234391597 2011-07-21 07:46:31,874 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: No previous transition plan was found (or we are ignoring an existing plan) for test1,282139,1311216321932.ffe0f4ab26eaa1796d2f99565ff24012. so generated a random one; hri=test1,282139,1311216321932.ffe0f4ab26eaa1796d2f99565ff24012., src=, dest=158-1-91-103,20020,1311232056655; 3 (online=3, exclude=null) available servers 2011-07-21 07:46:31,874 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Assigning region test1,282139,1311216321932.ffe0f4ab26eaa1796d2f99565ff24012. to 158-1-91-103,20020,1311232056655 2011-07-21 07:46:32,061 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_OPENED, server=158-1-91-103,20020,1311232056655, region=ffe0f4ab26eaa1796d2f99565ff24012 2011-07-21 07:46:32,493 DEBUG org.apache.hadoop.hbase.master.handler.OpenedRegionHandler: Handling OPENED event for ffe0f4ab26eaa1796d2f99565ff24012; deleting unassigned node 2011-07-21 07:46:32,493 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:20000-0x1314ac5addb0042-0x1314ac5addb0042 Deleting existing unassigned node for ffe0f4ab26eaa1796d2f99565ff24012 that is in expected state RS_ZK_REGION_OPENED 2011-07-21 07:46:32,693 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:20000-0x1314ac5addb0042-0x1314ac5addb0042 Successfully deleted unassigned node for region ffe0f4ab26eaa1796d2f99565ff24012 in expected state RS_ZK_REGION_OPENED 2011-07-21 07:46:32,717 DEBUG org.apache.hadoop.hbase.master.handler.OpenedRegionHandler: Opened region test1,282139,1311216321932.ffe0f4ab26eaa1796d2f99565ff24012. on 158-1-91-103,20020,1311232056655 105RS 2011-07-21 07:02:32,358 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing close of test1,282139,1311216321932.ffe0f4ab26eaa1796d2f99565ff24012. 2011-07-21 07:02:32,358 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing test1,282139,1311216321932.ffe0f4ab26eaa1796d2f99565ff24012.: disabling compactions & flushes 2011-07-21 07:02:32,358 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region test1,282139,1311216321932.ffe0f4ab26eaa1796d2f99565ff24012. 2011-07-21 07:02:32,358 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed test1,282139,1311216321932.ffe0f4ab26eaa1796d2f99565ff24012. 2011-07-21 07:02:32,358 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed region test1,282139,1311216321932.ffe0f4ab26eaa1796d2f99565ff24012. 2011-07-21 07:05:48,091 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: test1,282139,1311216321932.ffe0f4ab26eaa1796d2f99565ff24012. 2011-07-21 07:08:44,043 DEBUG org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Processing open of test1,282139,1311216321932.ffe0f4ab26eaa1796d2f99565ff24012. 2011-07-21 07:08:44,043 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x2314ac6763f0027 Attempting to transition node ffe0f4ab26eaa1796d2f99565ff24012 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2011-07-21 07:08:44,106 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x2314ac6763f0027 Successfully transitioned node ffe0f4ab26eaa1796d2f99565ff24012 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2011-07-21 07:08:44,106 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region: REGION => {NAME => 'test1,282139,1311216321932.ffe0f4ab26eaa1796d2f99565ff24012.', STARTKEY => '282139', ENDKEY => '282142', ENCODED => ffe0f4ab26eaa1796d2f99565ff24012, TABLE => {{NAME => 'test1', FAMILIES => [{NAME => 'value', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESSION => 'GZ', TTL => '2592000', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}} 2011-07-21 07:08:44,106 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Instantiated test1,282139,1311216321932.ffe0f4ab26eaa1796d2f99565ff24012. 2011-07-21 07:08:44,144 INFO org.apache.hadoop.hbase.regionserver.HRegion: Onlined test1,282139,1311216321932.ffe0f4ab26eaa1796d2f99565ff24012.; next sequenceid=1 2011-07-21 07:08:44,144 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x2314ac6763f0027 Attempting to transition node ffe0f4ab26eaa1796d2f99565ff24012 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2011-07-21 07:08:44,202 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x2314ac6763f0027 Successfully transitioned node ffe0f4ab26eaa1796d2f99565ff24012 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2011-07-21 07:08:44,232 INFO org.apache.hadoop.hbase.catalog.MetaEditor: Updated row test1,282139,1311216321932.ffe0f4ab26eaa1796d2f99565ff24012. in region .META.,,1 with server=158-1-91-105:20020, startcode=1311231874887 2011-07-21 07:08:44,232 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x2314ac6763f0027 Attempting to transition node ffe0f4ab26eaa1796d2f99565ff24012 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2011-07-21 07:08:44,291 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x2314ac6763f0027 Successfully transitioned node ffe0f4ab26eaa1796d2f99565ff24012 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2011-07-21 07:08:44,291 DEBUG org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Opened test1,282139,1311216321932.ffe0f4ab26eaa1796d2f99565ff24012. 2011-07-21 07:30:26,579 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: test1,282139,1311216321932.ffe0f4ab26eaa1796d2f99565ff24012. 2011-07-21 07:30:26,579 DEBUG org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Processing open of test1,282139,1311216321932.ffe0f4ab26eaa1796d2f99565ff24012. 2011-07-21 07:30:26,579 WARN org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Attempted open of test1,282139,1311216321932.ffe0f4ab26eaa1796d2f99565ff24012. but already online on this server 2011-07-21 07:42:49,046 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: test1,282139,1311216321932.ffe0f4ab26eaa1796d2f99565ff24012. 2011-07-21 07:42:49,046 DEBUG org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Processing open of test1,282139,1311216321932.ffe0f4ab26eaa1796d2f99565ff24012. 2011-07-21 07:42:49,046 WARN org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Attempted open of test1,282139,1311216321932.ffe0f4ab26eaa1796d2f99565ff24012. but already online on this server 103RS 2011-07-21 03:36:13,618 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: test1,282139,1311216321932.ffe0f4ab26eaa1796d2f99565ff24012. 2011-07-21 03:37:34,277 DEBUG org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Processing open of test1,282139,1311216321932.ffe0f4ab26eaa1796d2f99565ff24012. 2011-07-21 03:37:34,277 INFO org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Server stopping or stopped, skipping open of test1,282139,1311216321932.ffe0f4ab26eaa1796d2f99565ff24012. 2011-07-21 07:46:31,874 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: test1,282139,1311216321932.ffe0f4ab26eaa1796d2f99565ff24012. 2011-07-21 07:46:31,874 DEBUG org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Processing open of test1,282139,1311216321932.ffe0f4ab26eaa1796d2f99565ff24012. 2011-07-21 07:46:31,874 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x2314ac6763f0029 Attempting to transition node ffe0f4ab26eaa1796d2f99565ff24012 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2011-07-21 07:46:31,921 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x2314ac6763f0029 Successfully transitioned node ffe0f4ab26eaa1796d2f99565ff24012 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2011-07-21 07:46:31,921 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region: REGION => {NAME => 'test1,282139,1311216321932.ffe0f4ab26eaa1796d2f99565ff24012.', STARTKEY => '282139', ENDKEY => '282142', ENCODED => ffe0f4ab26eaa1796d2f99565ff24012, TABLE => {{NAME => 'test1', FAMILIES => [{NAME => 'value', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESSION => 'GZ', TTL => '2592000', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}} 2011-07-21 07:46:31,921 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Instantiated test1,282139,1311216321932.ffe0f4ab26eaa1796d2f99565ff24012. 2011-07-21 07:46:31,927 INFO org.apache.hadoop.hbase.regionserver.HRegion: Onlined test1,282139,1311216321932.ffe0f4ab26eaa1796d2f99565ff24012.; next sequenceid=1 2011-07-21 07:46:31,928 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x2314ac6763f0029 Attempting to transition node ffe0f4ab26eaa1796d2f99565ff24012 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2011-07-21 07:46:31,968 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x2314ac6763f0029 Successfully transitioned node ffe0f4ab26eaa1796d2f99565ff24012 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2011-07-21 07:46:31,969 INFO org.apache.hadoop.hbase.catalog.MetaEditor: Updated row test1,282139,1311216321932.ffe0f4ab26eaa1796d2f99565ff24012. in region .META.,,1 with server=158-1-91-103:20020, startcode=1311232056655 2011-07-21 07:46:31,969 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x2314ac6763f0029 Attempting to transition node ffe0f4ab26eaa1796d2f99565ff24012 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2011-07-21 07:46:32,009 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x2314ac6763f0029 Successfully transitioned node ffe0f4ab26eaa1796d2f99565ff24012 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2011-07-21 07:46:32,009 DEBUG org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Opened test1,282139,1311216321932.ffe0f4ab26eaa1796d2f99565ff24012. 101HM 2011-07-21 07:07:00,902 INFO org.apache.hadoop.hbase.master.ServerManager: Registering server=158-1-91-105,20020,1311231874887, regionCount=971, userLoad=true 2011-07-21 07:07:01,569 INFO org.apache.hadoop.hbase.master.ServerManager: Registering server=158-1-91-101,20020,1311231878544, regionCount=951, userLoad=true 2011-07-21 07:07:36,903 INFO org.apache.hadoop.hbase.master.ServerManager: Registering server=158-1-91-103,20020,1311232056655, regionCount=0, userLoad=false