2015-08-31 18:01:05,402 DEBUG [RegionOpenAndInitThread-NB_APP_BEHAVIOR_LABEL_000000-6] regionserver.HRegion(658): Instantiated NB_APP_BEHAVIOR_LABEL_000000,F\x180100\x18,1441015264846.2070d83bfd7c3fa6950c859ce842039e. 2015-08-31 18:01:05,403 DEBUG [RegionOpenAndInitThread-NB_APP_BEHAVIOR_LABEL_000000-6] regionserver.HRegion(1141): Closing NB_APP_BEHAVIOR_LABEL_000000,F\x180100\x18,1441015264846.2070d83bfd7c3fa6950c859ce842039e.: disabling compactions & flushes 2015-08-31 18:01:05,403 DEBUG [RegionOpenAndInitThread-NB_APP_BEHAVIOR_LABEL_000000-6] regionserver.HRegion(1168): Updates disabled for region NB_APP_BEHAVIOR_LABEL_000000,F\x180100\x18,1441015264846.2070d83bfd7c3fa6950c859ce842039e. 2015-08-31 18:01:05,403 INFO [RegionOpenAndInitThread-NB_APP_BEHAVIOR_LABEL_000000-6] regionserver.HRegion(1250): Closed NB_APP_BEHAVIOR_LABEL_000000,F\x180100\x18,1441015264846.2070d83bfd7c3fa6950c859ce842039e. 2015-08-31 18:01:05,619 DEBUG [hm,60000,1438368907764-GeneralBulkAssigner-6] zookeeper.ZKAssign(175): master:60000-0x24ee432542d01eb, quorum=hs5:2181,hs4:2181,hm:2181, baseZNode=/hbase Async create of unassigned node 2070d83bfd7c3fa6950c859ce842039e with OFFLINE state 2015-08-31 18:01:05,647 DEBUG [main-EventThread] master.OfflineCallback(69): rs={2070d83bfd7c3fa6950c859ce842039e state=OFFLINE, ts=1441015265616, server=null}, server=hs15,60020,1438578735538 2015-08-31 18:01:05,650 DEBUG [main-EventThread] master.OfflineCallback$ExistCallback(106): rs={2070d83bfd7c3fa6950c859ce842039e state=OFFLINE, ts=1441015265616, server=null}, server=hs15,60020,1438578735538 2015-08-31 18:01:05,651 INFO [hm,60000,1438368907764-GeneralBulkAssigner-6] master.RegionStates(826): Transition {2070d83bfd7c3fa6950c859ce842039e state=OFFLINE, ts=1441015265619, server=null} to {2070d83bfd7c3fa6950c859ce842039e state=PENDING_OPEN, ts=1441015265651, server=hs15,60020,1438578735538} 2015-08-31 18:01:05,977 DEBUG [AM.ZK.Worker-pool2-t3644] master.AssignmentManager(926): Handling RS_ZK_REGION_OPENING, server=hs15,60020,1438578735538, region=2070d83bfd7c3fa6950c859ce842039e, current_state={2070d83bfd7c3fa6950c859ce842039e state=PENDING_OPEN, ts=1441015265651, server=hs15,60020,1438578735538} 2015-08-31 18:01:05,978 INFO [AM.ZK.Worker-pool2-t3644] master.RegionStates(826): Transition {2070d83bfd7c3fa6950c859ce842039e state=PENDING_OPEN, ts=1441015265651, server=hs15,60020,1438578735538} to {2070d83bfd7c3fa6950c859ce842039e state=OPENING, ts=1441015265978, server=hs15,60020,1438578735538} 2015-08-31 18:01:06,037 DEBUG [AM.ZK.Worker-pool2-t3654] master.AssignmentManager(926): Handling RS_ZK_REGION_OPENED, server=hs15,60020,1438578735538, region=2070d83bfd7c3fa6950c859ce842039e, current_state={2070d83bfd7c3fa6950c859ce842039e state=OPENING, ts=1441015265978, server=hs15,60020,1438578735538} 2015-08-31 18:01:06,037 INFO [AM.ZK.Worker-pool2-t3654] master.RegionStates(826): Transition {2070d83bfd7c3fa6950c859ce842039e state=OPENING, ts=1441015265978, server=hs15,60020,1438578735538} to {2070d83bfd7c3fa6950c859ce842039e state=OPEN, ts=1441015266037, server=hs15,60020,1438578735538} 2015-08-31 18:01:06,037 DEBUG [AM.ZK.Worker-pool2-t3654] handler.OpenedRegionHandler(149): Handling OPENED of 2070d83bfd7c3fa6950c859ce842039e from hs15,60020,1438578735538; deleting unassigned node 2015-08-31 18:01:06,075 DEBUG [AM.ZK.Worker-pool2-t3654] zookeeper.ZKAssign(480): master:60000-0x24ee432542d01eb, quorum=hs5:2181,hs4:2181,hm:2181, baseZNode=/hbase Deleted unassigned node 2070d83bfd7c3fa6950c859ce842039e in expected state RS_ZK_REGION_OPENED 2015-08-31 18:01:06,097 DEBUG [AM.ZK.Worker-pool2-t3655] master.AssignmentManager(1312): Znode NB_APP_BEHAVIOR_LABEL_000000,F\x180100\x18,1441015264846.2070d83bfd7c3fa6950c859ce842039e. deleted, state: {2070d83bfd7c3fa6950c859ce842039e state=OPEN, ts=1441015266037, server=hs15,60020,1438578735538} 2015-08-31 18:01:06,098 INFO [AM.ZK.Worker-pool2-t3655] master.RegionStates(371): Onlined 2070d83bfd7c3fa6950c859ce842039e on hs15,60020,1438578735538 2015-09-03 00:56:40,825 DEBUG [hm,60000,1438368907764-BalancerChore] regionserver.HRegionFileSystem(192): No StoreFiles for: hdfs://ngpcluster/hbase/data/default/NB_APP_BEHAVIOR_LABEL_000000/2070d83bfd7c3fa6950c859ce842039e/D 2015-09-06 00:46:39,959 DEBUG [hm,60000,1438368907764-BalancerChore] regionserver.HRegionFileSystem(192): No StoreFiles for: hdfs://ngpcluster/hbase/data/default/NB_APP_BEHAVIOR_LABEL_000000/2070d83bfd7c3fa6950c859ce842039e/D 2015-09-06 01:26:41,382 DEBUG [hm,60000,1438368907764-BalancerChore] regionserver.HRegionFileSystem(192): No StoreFiles for: hdfs://ngpcluster/hbase/data/default/NB_APP_BEHAVIOR_LABEL_000000/2070d83bfd7c3fa6950c859ce842039e/D 2015-09-06 01:34:58,989 INFO [MASTER_SERVER_OPERATIONS-hm:60000-2] master.RegionStates(826): Transition {2070d83bfd7c3fa6950c859ce842039e state=OPEN, ts=1441015266098, server=hs15,60020,1438578735538} to {2070d83bfd7c3fa6950c859ce842039e state=OFFLINE, ts=1441474498989, server=hs15,60020,1438578735538} 2015-09-06 01:34:58,989 INFO [MASTER_SERVER_OPERATIONS-hm:60000-2] master.RegionStates(467): Offlined 2070d83bfd7c3fa6950c859ce842039e from hs15,60020,1438578735538 2015-09-06 01:34:59,138 DEBUG [hm,60000,1438368907764-GeneralBulkAssigner-19] zookeeper.ZKAssign(175): master:60000-0x24ee432542d01eb, quorum=hs5:2181,hs4:2181,hm:2181, baseZNode=/hbase Async create of unassigned node 2070d83bfd7c3fa6950c859ce842039e with OFFLINE state 2015-09-06 01:34:59,227 DEBUG [main-EventThread] master.OfflineCallback(69): rs={2070d83bfd7c3fa6950c859ce842039e state=OFFLINE, ts=1441474498989, server=hs15,60020,1438578735538}, server=hs4,60020,1441213185092 2015-09-06 01:34:59,300 DEBUG [main-EventThread] master.OfflineCallback$ExistCallback(106): rs={2070d83bfd7c3fa6950c859ce842039e state=OFFLINE, ts=1441474498989, server=hs15,60020,1438578735538}, server=hs4,60020,1441213185092 2015-09-06 01:34:59,424 INFO [hm,60000,1438368907764-GeneralBulkAssigner-19] master.RegionStates(826): Transition {2070d83bfd7c3fa6950c859ce842039e state=OFFLINE, ts=1441474499138, server=hs15,60020,1438578735538} to {2070d83bfd7c3fa6950c859ce842039e state=PENDING_OPEN, ts=1441474499424, server=hs4,60020,1441213185092} 2015-09-06 01:35:59,521 DEBUG [hm,60000,1438368907764-GeneralBulkAssigner-19] master.AssignmentManager(1768): Bulk assigner openRegion() to hs4,60020,1441213185092 has timed out, but the regions might already be opened on it. java.net.SocketTimeoutException: Call to hs4/15.173.0.115:60020 failed because java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/15.173.0.110:33771 remote=hs4/15.173.0.115:60020] at org.apache.hadoop.hbase.ipc.RpcClient.wrapException(RpcClient.java:1486) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1461) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.openRegion(AdminProtos.java:20964) at org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:709) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1722) at org.apache.hadoop.hbase.master.GeneralBulkAssigner$SingleServerBulkAssigner.run(GeneralBulkAssigner.java:228) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/15.173.0.110:33771 remote=hs4/15.173.0.115:60020] at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) at java.io.FilterInputStream.read(FilterInputStream.java:133) at java.io.FilterInputStream.read(FilterInputStream.java:133) at org.apache.hadoop.hbase.ipc.RpcClient$Connection$PingInputStream.read(RpcClient.java:558) at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) at java.io.BufferedInputStream.read(BufferedInputStream.java:254) at java.io.DataInputStream.readInt(DataInputStream.java:387) at org.apache.hadoop.hbase.ipc.RpcClient$Connection.readResponse(RpcClient.java:1076) at org.apache.hadoop.hbase.ipc.RpcClient$Connection.run(RpcClient.java:727) 2015-09-06 01:36:07,063 DEBUG [hm,60000,1438368907764-GeneralBulkAssigner-19] master.AssignmentManager(2293): ALREADY_OPENED NB_APP_BEHAVIOR_LABEL_000000,F\x180100\x18,1441015264846.2070d83bfd7c3fa6950c859ce842039e. to hs4,60020,1441213185092 2015-09-06 01:36:07,073 WARN [hm,60000,1438368907764-GeneralBulkAssigner-19] zookeeper.ZKAssign(458): master:60000-0x24ee432542d01eb, quorum=hs5:2181,hs4:2181,hm:2181, baseZNode=/hbase Attempting to delete unassigned node 2070d83bfd7c3fa6950c859ce842039e in M_ZK_REGION_OFFLINE state but node is in RS_ZK_REGION_OPENED state 2015-09-06 01:36:07,073 INFO [hm,60000,1438368907764-GeneralBulkAssigner-19] master.AssignmentManager(3614): Failed to delete the offline node for 2070d83bfd7c3fa6950c859ce842039e. The node type may not match 2015-09-06 01:36:07,073 INFO [hm,60000,1438368907764-GeneralBulkAssigner-19] master.RegionStates(826): Transition {2070d83bfd7c3fa6950c859ce842039e state=PENDING_OPEN, ts=1441474499424, server=hs4,60020,1441213185092} to {2070d83bfd7c3fa6950c859ce842039e state=OPEN, ts=1441474567073, server=hs4,60020,1441213185092} 2015-09-06 01:36:07,073 INFO [hm,60000,1438368907764-GeneralBulkAssigner-19] master.RegionStates(371): Onlined 2070d83bfd7c3fa6950c859ce842039e on hs4,60020,1441213185092 2015-09-06 01:36:33,960 DEBUG [AM.ZK.Worker-pool2-t5251] master.AssignmentManager(926): Handling RS_ZK_REGION_OPENED, server=hs4,60020,1441213185092, region=2070d83bfd7c3fa6950c859ce842039e, which is more than 15 seconds late, current_state={2070d83bfd7c3fa6950c859ce842039e state=OPEN, ts=1441474567073, server=hs4,60020,1441213185092} 2015-09-06 01:36:33,961 WARN [AM.ZK.Worker-pool2-t5251] master.AssignmentManager(1061): Received OPENED for 2070d83bfd7c3fa6950c859ce842039e from hs4,60020,1441213185092 but the region isn't PENDING_OPEN/OPENING here: {2070d83bfd7c3fa6950c859ce842039e state=OPEN, ts=1441474567073, server=hs4,60020,1441213185092} 2015-09-06 01:36:33,965 DEBUG [AM.ZK.Worker-pool2-t5251] master.AssignmentManager(1849): Sent CLOSE to hs4,60020,1441213185092 for region NB_APP_BEHAVIOR_LABEL_000000,F\x180100\x18,1441015264846.2070d83bfd7c3fa6950c859ce842039e.