2010-12-10 08:49:51,629 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: hlog file splitting completed in 9566 ms for hdfs://haus01.sf.cloudera.com:11020/hbase-normal/.logs/haus02.sf.cloudera.com,60020,1291950999086 2010-12-10 08:49:51,629 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x12c780205200058 Creating (or updating) unassigned node for 1028785192 with OFFLINE state 2010-12-10 08:49:51,648 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: master:60000-0x12c780205200058 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/unassigned/1028785192 2010-12-10 08:49:51,649 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: No previous transition plan was found (or we are ignoring an existing plan) for .META.,,1.1028785192 so generated a random one; hri=.META.,,1.1028785192, src=, dest=haus05.sf.cloudera.com,60020,1291950999807; 4 (online=4, exclude=null) available servers 2010-12-10 08:49:51,649 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Assigning region .META.,,1.1028785192 to haus05.sf.cloudera.com,60020,1291950999807 2010-12-10 08:49:51,649 DEBUG org.apache.hadoop.hbase.zookeeper.ZKUtil: master:60000-0x12c780205200058 Retrieved 51 byte(s) of data from znode /hbase/unassigned/1028785192 and set watcher; region=.META.,,1, server=haus01.sf.cloudera.com:60000, state=M_ZK_REGION_OFFLINE 2010-12-10 08:49:51,649 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=M_ZK_REGION_OFFLINE, server=haus01.sf.cloudera.com:60000, region=1028785192/.META. 2010-12-10 08:49:51,649 DEBUG org.apache.hadoop.hbase.zookeeper.ZKUtil: master:60000-0x12c780205200058 Retrieved 51 byte(s) of data from znode /hbase/unassigned/1028785192 and set watcher; region=.META.,,1, server=haus01.sf.cloudera.com:60000, state=M_ZK_REGION_OFFLINE 2010-12-10 08:49:51,649 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: master:60000-0x12c780205200058 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/unassigned 2010-12-10 08:49:51,665 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: master:60000-0x12c780205200058 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/unassigned/1028785192 2010-12-10 08:49:51,665 DEBUG org.apache.hadoop.hbase.zookeeper.ZKUtil: master:60000-0x12c780205200058 Retrieved 65 byte(s) of data from znode /hbase/unassigned/1028785192 and set watcher; region=.META.,,1, server=haus05.sf.cloudera.com,60020,1291950999807, state=RS_ZK_REGION_OPENING 2010-12-10 08:49:51,665 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_OPENING, server=haus05.sf.cloudera.com,60020,1291950999807, region=1028785192/.META. 2010-12-10 08:49:51,666 DEBUG org.apache.hadoop.hbase.zookeeper.ZKUtil: master:60000-0x12c780205200058 Retrieved 65 byte(s) of data from znode /hbase/unassigned/1028785192 and set watcher; region=.META.,,1, server=haus05.sf.cloudera.com,60020,1291950999807, state=RS_ZK_REGION_OPENING 2010-12-10 08:49:51,731 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: master:60000-0x12c780205200058 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/unassigned/1028785192 2010-12-10 08:49:51,732 DEBUG org.apache.hadoop.hbase.zookeeper.ZKUtil: master:60000-0x12c780205200058 Retrieved 65 byte(s) of data from znode /hbase/unassigned/1028785192 and set watcher; region=.META.,,1, server=haus05.sf.cloudera.com,60020,1291950999807, state=RS_ZK_REGION_OPENING 2010-12-10 08:49:51,732 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_OPENING, server=haus05.sf.cloudera.com,60020,1291950999807, region=1028785192/.META. 2010-12-10 08:49:51,732 DEBUG org.apache.hadoop.hbase.zookeeper.ZKUtil: master:60000-0x12c780205200058 Retrieved 65 byte(s) of data from znode /hbase/unassigned/1028785192 and set watcher; region=.META.,,1, server=haus05.sf.cloudera.com,60020,1291950999807, state=RS_ZK_REGION_OPENING 2010-12-10 08:49:51,873 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: master:60000-0x12c780205200058 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/unassigned/1028785192 2010-12-10 08:49:51,873 DEBUG org.apache.hadoop.hbase.zookeeper.ZKUtil: master:60000-0x12c780205200058 Retrieved 65 byte(s) of data from znode /hbase/unassigned/1028785192 and set watcher; region=.META.,,1, server=haus05.sf.cloudera.com,60020,1291950999807, state=RS_ZK_REGION_OPENED 2010-12-10 08:49:51,873 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_OPENED, server=haus05.sf.cloudera.com,60020,1291950999807, region=1028785192/.META. 2010-12-10 08:49:51,874 DEBUG org.apache.hadoop.hbase.master.handler.OpenedRegionHandler: Handling OPENED event for 1028785192; deleting unassigned node 2010-12-10 08:49:51,874 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x12c780205200058 Deleting existing unassigned node for 1028785192 that is in expected state RS_ZK_REGION_OPENED 2010-12-10 08:49:51,874 DEBUG org.apache.hadoop.hbase.zookeeper.ZKUtil: master:60000-0x12c780205200058 Retrieved 65 byte(s) of data from znode /hbase/unassigned/1028785192 and set watcher; region=.META.,,1, server=haus05.sf.cloudera.com,60020,1291950999807, state=RS_ZK_REGION_OPENED 2010-12-10 08:49:51,874 DEBUG org.apache.hadoop.hbase.zookeeper.ZKUtil: master:60000-0x12c780205200058 Retrieved 65 byte(s) of data from znode /hbase/unassigned/1028785192; data=region=.META.,,1, server=haus05.sf.cloudera.com,60020,1291950999807, state=RS_ZK_REGION_OPENED 2010-12-10 08:49:51,881 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: master:60000-0x12c780205200058 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/unassigned/1028785192 2010-12-10 08:49:51,881 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x12c780205200058 Successfully deleted unassigned node for region 1028785192 in expected state RS_ZK_REGION_OPENED 2010-12-10 08:49:51,881 DEBUG org.apache.hadoop.hbase.master.handler.OpenedRegionHandler: Opened region .META.,,1.1028785192 on haus05.sf.cloudera.com,60020,1291950999807 2010-12-10 08:49:51,881 DEBUG org.apache.hadoop.hbase.zookeeper.ZKUtil: master:60000-0x12c780205200058 Set watcher on existing znode /hbase/unassigned/1028785192 2010-12-10 08:49:51,881 INFO org.apache.hadoop.hbase.zookeeper.MetaNodeTracker: Detected completed assignment of META, notifying catalog tracker 2010-12-10 08:50:24,088 DEBUG org.apache.hadoop.hbase.master.HMaster: Not running balancer because processing dead regionserver(s): [haus02.sf.cloudera.com,60020,1291950999086, haus03.sf.cloudera.com,60020,1291950998814] 2010-12-10 08:50:47,848 INFO org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Received exception accessing META during server shutdown of haus02.sf.cloudera.com,60020,1291950999086, retrying META read 2010-12-10 08:50:48,850 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@6b5d49a1; hsa=haus05.sf.cloudera.com:60020 2010-12-10 08:50:48,850 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Cache hit for row <> in tableName .META.: location server haus02.sf.cloudera.com:60020, location region name .META.,,1.1028785192 2010-12-10 08:50:48,850 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Removed .META.,,1.1028785192 for tableName=.META. from cache because of 2010-12-10 08:50:48,853 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Cached location for .META.,,1.1028785192 is haus05.sf.cloudera.com:60020 2010-12-10 08:51:47,855 WARN org.apache.hadoop.hbase.master.CatalogJanitor: Failed scan of catalog table java.net.SocketTimeoutException: Call to /172.29.5.33:60020 failed on socket timeout exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/172.29.5.32:52396 remote=/172.29.5.33:60020] at org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:784) at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:757) at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257) at $Proxy7.getRegionInfo(Unknown Source) at org.apache.hadoop.hbase.catalog.CatalogTracker.verifyRegionLocation(CatalogTracker.java:416) at org.apache.hadoop.hbase.catalog.CatalogTracker.getMetaServerConnection(CatalogTracker.java:270) at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:322) at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMetaServerConnectionDefault(CatalogTracker.java:362) at org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:237) at org.apache.hadoop.hbase.master.CatalogJanitor.scan(CatalogJanitor.java:108) at org.apache.hadoop.hbase.master.CatalogJanitor.chore(CatalogJanitor.java:77) at org.apache.hadoop.hbase.Chore.run(Chore.java:66) Caused by: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/172.29.5.32:52396 remote=/172.29.5.33:60020] at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128) at java.io.FilterInputStream.read(FilterInputStream.java:116) at org.apache.hadoop.hbase.ipc.HBaseClient$Connection$PingInputStream.read(HBaseClient.java:281) at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) at java.io.BufferedInputStream.read(BufferedInputStream.java:237) at java.io.DataInputStream.readInt(DataInputStream.java:370) at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:521) at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:459) 2010-12-10 08:52:39,433 DEBUG org.apache.hadoop.hbase.client.MetaScanner: Scanning .META. starting at row= for max=2147483647 rows 2010-12-10 08:52:39,433 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Cache hit for row <> in tableName .META.: location server haus05.sf.cloudera.com:60020, location region name .META.,,1.1028785192 2010-12-10 08:52:40,667 DEBUG org.apache.hadoop.hbase.client.MetaScanner: Scanning .META. starting at row= for max=2147483647 rows 2010-12-10 08:52:40,667 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Cache hit for row <> in tableName .META.: location server haus05.sf.cloudera.com:60020, location region name .META.,,1.1028785192 2010-12-10 08:52:41,471 DEBUG org.apache.hadoop.hbase.client.MetaScanner: Scanning .META. starting at row= for max=2147483647 rows 2010-12-10 08:52:41,471 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Cache hit for row <> in tableName .META.: location server haus05.sf.cloudera.com:60020, location region name .META.,,1.1028785192 2010-12-10 08:52:47,854 WARN org.apache.hadoop.hbase.zookeeper.MetaNodeTracker: Tried to reset META server location after seeing the completion of a new META assignment but got an IOE java.net.SocketTimeoutException: Call to /172.29.5.33:60020 failed on socket timeout exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/172.29.5.32:52401 remote=/172.29.5.33:60020] at org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:784) at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:757) at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257) at $Proxy7.getRegionInfo(Unknown Source) at org.apache.hadoop.hbase.catalog.CatalogTracker.verifyRegionLocation(CatalogTracker.java:416) at org.apache.hadoop.hbase.catalog.CatalogTracker.getMetaServerConnection(CatalogTracker.java:270) at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:322) at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMetaServerConnectionDefault(CatalogTracker.java:362) at org.apache.hadoop.hbase.zookeeper.MetaNodeTracker.nodeDeleted(MetaNodeTracker.java:64) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:261) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Caused by: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/172.29.5.32:52401 remote=/172.29.5.33:60020] at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128) at java.io.FilterInputStream.read(FilterInputStream.java:116) at org.apache.hadoop.hbase.ipc.HBaseClient$Connection$PingInputStream.read(HBaseClient.java:281) at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) at java.io.BufferedInputStream.read(BufferedInputStream.java:237) at java.io.DataInputStream.readInt(DataInputStream.java:370) at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:521) at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:459) 2010-12-10 08:52:47,856 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: master:60000-0x12c780205200058 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/unassigned 2010-12-10 08:53:47,860 INFO org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Received exception accessing META during server shutdown of haus02.sf.cloudera.com,60020,1291950999086, retrying META read