2010-12-10 10:20:06,698 INFO org.apache.hadoop.hbase.master.LoadBalancer: Calculated a load balance in 0ms. Moving 6 regions off of 4 overloaded servers onto 1 less loaded servers 2010-12-10 10:20:06,698 INFO org.apache.hadoop.hbase.master.HMaster: balance hri=usertable,user876353700,1291843173343.182d193c6317cab6486eab93bf95b6a7., src=haus05.sf.cloudera.com,60020,1291950999807, dest=haus02.sf.cloudera.com,60020,1292005167248 2010-12-10 10:20:06,699 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of region usertable,user876353700,1291843173343.182d193c6317cab6486eab93bf95b6a7. (offlining) 2010-12-10 10:20:06,700 INFO org.apache.hadoop.hbase.master.AssignmentManager: Failed connect to serverName=haus05.sf.cloudera.com,60020,1291950999807, load=(requests=67, regions=7, usedHeap=863, maxHeap=8185), message=Connection refused, region=182d193c6317cab6486eab93bf95b6a7 .... 2010-12-10 10:20:36,018 INFO org.apache.hadoop.hbase.zookeeper.RegionServerTracker: RegionServer ephemeral node deleted, processing expiration [haus05.sf.cloudera.com,60020,1291950999807] 2010-12-10 10:20:36,018 DEBUG org.apache.hadoop.hbase.master.ServerManager: Added=haus05.sf.cloudera.com,60020,1291950999807 to dead servers, submitted shutdown handler to be executed, root=true, meta=true 2010-12-10 10:20:36,018 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: master:60000-0x12c780205200058 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2010-12-10 10:20:36,018 INFO org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Splitting logs for haus05.sf.cloudera.com,60020,1291950999807 ... 2010-12-10 10:20:39,059 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: usertable,user876353700,1291843173343.182d193c6317cab6486eab93bf95b6a7. state=PENDING_CLOSE, ts=1292005206699 2010-12-10 10:20:39,059 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been PENDING_CLOSE for too long, running forced unassign again on region=usertable,user876353700,1291843173343.182d193c6317cab6486eab93bf95b6a7. 2010-12-10 10:20:39,059 DEBUG org.apache.hadoop.hbase.zookeeper.ZKUtil: master:60000-0x12c780205200058 Set watcher on existing znode /hbase/unassigned/182d193c6317cab6486eab93bf95b6a7 2010-12-10 10:20:39,059 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of region usertable,user876353700,1291843173343.182d193c6317cab6486eab93bf95b6a7. (offlining) 2010-12-10 10:20:39,059 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Attempting to unassign region usertable,user876353700,1291843173343.182d193c6317cab6486eab93bf95b6a7. which is already pending close but forcing an additional close 2010-12-10 10:20:39,060 DEBUG org.apache.hadoop.hbase.master.ServerManager: New connection to haus05.sf.cloudera.com,60020,1291950999807 2010-12-10 10:20:39,060 INFO org.apache.hadoop.hbase.master.AssignmentManager: Failed connect to serverName=haus05.sf.cloudera.com,60020,1291950999807, load=(requests=67, regions=7, usedHeap=863, maxHeap=8185), message=Connection refused, region=182d193c6317cab6486eab93bf95b6a7 ... 2010-12-10 10:21:09,093 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: usertable,user876353700,1291843173343.182d193c6317cab6486eab93bf95b6a7. state=PENDING_CLOSE, ts=1292005239060 2010-12-10 10:21:09,093 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been PENDING_CLOSE for too long, running forced unassign again on region=usertable,user876353700,1291843173343.182d193c6317cab6486eab93bf95b6a7. 2010-12-10 10:21:09,595 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=470435 entries from hdfs://haus01.sf.cloudera.com:11020/hbase-normal/.logs/haus05.sf.cloudera.com,60020,1291950999807/haus05.sf.cloudera.com%3A60020.1292004469117 2010-12-10 10:21:09,595 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog 1 of 16: hdfs://haus01.sf.cloudera.com:11020/hbase-normal/.logs/haus05.sf.cloudera.com,60020,1291950999807/haus05.sf.cloudera.com%3A60020.1292004829636, length=0 2010-12-10 10:21:09,595 INFO org.apache.hadoop.hbase.util.FSUtils: Recovering file hdfs://haus01.sf.cloudera.com:11020/hbase-normal/.logs/haus05.sf.cloudera.com,60020,1291950999807/haus05.sf.cloudera.com%3A60020.1292004829636 2010-12-10 10:21:09,893 DEBUG org.apache.hadoop.hbase.zookeeper.ZKUtil: master:60000-0x12c780205200058 Set watcher on existing znode /hbase/unassigned/182d193c6317cab6486eab93bf95b6a7 2010-12-10 10:21:09,893 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of region usertable,user876353700,1291843173343.182d193c6317cab6486eab93bf95b6a7. (offlining) 2010-12-10 10:21:09,893 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Attempting to unassign region usertable,user876353700,1291843173343.182d193c6317cab6486eab93bf95b6a7. which is already pending close but forcing an additional close 2010-12-10 10:21:09,895 INFO org.apache.hadoop.hbase.master.AssignmentManager: Failed connect to serverName=haus05.sf.cloudera.com,60020,1291950999807, load=(requests=67, regions=7, usedHeap=863, maxHeap=8185), message=Connection refused, region=182d193c6317cab6486eab93bf95b6a7 ... eventually this in a loop every 10 seconds: 2010-12-10 10:22:49,117 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: usertable,user876353700,1291843173343.182d193c6317cab6486eab93bf95b6a7. state=PENDING_CLOSE, ts=1292005269893 2010-12-10 10:22:49,117 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been PENDING_CLOSE for too long, running forced unassign again on region=usertable,user876353700,1291843173343.182d193c6317cab6486eab93bf95b6a7. 2010-12-10 10:22:49,118 DEBUG org.apache.hadoop.hbase.zookeeper.ZKUtil: master:60000-0x12c780205200058 Set watcher on existing znode /hbase/unassigned/182d193c6317cab6486eab93bf95b6a7 2010-12-10 10:22:49,118 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of region usertable,user876353700,1291843173343.182d193c6317cab6486eab93bf95b6a7. (offlining) 2010-12-10 10:22:49,118 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Attempted to unassign region usertable,user876353700,1291843173343.182d193c6317cab6486eab93bf95b6a7. but it is not currently assigned anywhere