| Failedorg.apache.hadoop.hbase.util.TestHBaseFsck.testCheckTableLocks Failing for the past 3 builds
(Since #67 )
Error Messageexpected:<[EXPIRED_TABLE_LOCK]> but was:<[UNKNOWN, NO_META_REGION, RS_CONNECT_FAILURE, RS_CONNECT_FAILURE, RS_CONNECT_FAILURE, EXPIRED_TABLE_LOCK]> Stacktracejava.lang.AssertionError: expected:<[EXPIRED_TABLE_LOCK]> but was:<[UNKNOWN, NO_META_REGION, RS_CONNECT_FAILURE, RS_CONNECT_FAILURE, RS_CONNECT_FAILURE, EXPIRED_TABLE_LOCK]>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:144)
at org.apache.hadoop.hbase.util.hbck.HbckTestingUtil.assertErrors(HbckTestingUtil.java:96)
at org.apache.hadoop.hbase.util.TestHBaseFsck.testCheckTableLocks(TestHBaseFsck.java:2101)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
Standard OutputVersion: 0.98.0.2.1.1.0-154-hadoop2
Number of live region servers: 3
c64-s15.cs1cloud.internal,52297,1393848623312
c64-s15.cs1cloud.internal,53849,1393848623494
c64-s15.cs1cloud.internal,38669,1393848623652
Number of dead region servers: 0
Master: c64-s15.cs1cloud.internal,55261,1393848622551
Number of backup masters: 0
Average load: 0.6666666666666666
Number of requests: 1649
Number of regions: 2
Number of regions in transition: 0
Number of empty REGIONINFO_QUALIFIER rows in hbase:meta: 0
Number of Tables: 1
Table: hbase:namespace rw families: 1
---- Table 'hbase:meta': region split map
: [ { meta => hbase:meta,,1.1588230740, hdfs => hdfs://localhost:47748/user/jenkins/hbase/data/hbase/meta/1588230740, deployed => c64-s15.cs1cloud.internal,53849,1393848623494;hbase:meta,,1.1588230740 }, ]
null:
---- Table 'hbase:meta': overlap groups
There are 0 overlap groups with 0 overlapping regions
---- Table 'hbase:namespace': region split map
: [ { meta => hbase:namespace,,1393848627663.a62c7d597526da258a2019c6787e5887., hdfs => hdfs://localhost:47748/user/jenkins/hbase/data/hbase/namespace/a62c7d597526da258a2019c6787e5887, deployed => c64-s15.cs1cloud.internal,53849,1393848623494;hbase:namespace,,1393848627663.a62c7d597526da258a2019c6787e5887. }, ]
null:
---- Table 'hbase:namespace': overlap groups
There are 0 overlap groups with 0 overlapping regions
Summary:
hbase:meta is okay.
Number of regions: 1
Deployed on: c64-s15.cs1cloud.internal,53849,1393848623494
hbase:namespace is okay.
Number of regions: 1
Deployed on: c64-s15.cs1cloud.internal,53849,1393848623494
0 inconsistencies detected.
Status: OK
Version: 0.98.0.2.1.1.0-154-hadoop2
Number of live region servers: 3
c64-s15.cs1cloud.internal,52297,1393848623312
c64-s15.cs1cloud.internal,53849,1393848623494
c64-s15.cs1cloud.internal,38669,1393848623652
Number of dead region servers: 0
Master: c64-s15.cs1cloud.internal,55261,1393848622551
Number of backup masters: 0
Average load: 0.6666666666666666
Number of requests: 1657
Number of regions: 2
Number of regions in transition: 0
Number of empty REGIONINFO_QUALIFIER rows in hbase:meta: 0
Number of Tables: 1
Table: hbase:namespace rw families: 1
---- Table 'hbase:meta': region split map
: [ { meta => hbase:meta,,1.1588230740, hdfs => hdfs://localhost:47748/user/jenkins/hbase/data/hbase/meta/1588230740, deployed => c64-s15.cs1cloud.internal,53849,1393848623494;hbase:meta,,1.1588230740 }, ]
null:
---- Table 'hbase:meta': overlap groups
There are 0 overlap groups with 0 overlapping regions
---- Table 'hbase:namespace': region split map
: [ { meta => hbase:namespace,,1393848627663.a62c7d597526da258a2019c6787e5887., hdfs => hdfs://localhost:47748/user/jenkins/hbase/data/hbase/namespace/a62c7d597526da258a2019c6787e5887, deployed => c64-s15.cs1cloud.internal,53849,1393848623494;hbase:namespace,,1393848627663.a62c7d597526da258a2019c6787e5887. }, ]
null:
---- Table 'hbase:namespace': overlap groups
There are 0 overlap groups with 0 overlapping regions
Table lock acquire attempt found:[tableName=
defaultfoo, lockOwner=localhost,60000,1, threadId=2364, purpose=testCheckTableLocks, isShared=false, createTime=63]
Summary:
hbase:meta is okay.
Number of regions: 1
Deployed on: c64-s15.cs1cloud.internal,53849,1393848623494
hbase:namespace is okay.
Number of regions: 1
Deployed on: c64-s15.cs1cloud.internal,53849,1393848623494
0 inconsistencies detected.
Status: OK
Version: 0.98.0.2.1.1.0-154-hadoop2
Number of live region servers: 3
c64-s15.cs1cloud.internal,52297,1393848623312
c64-s15.cs1cloud.internal,53849,1393848623494
c64-s15.cs1cloud.internal,38669,1393848623652
Number of dead region servers: 0
Master: c64-s15.cs1cloud.internal,55261,1393848622551
Number of backup masters: 0
Average load: 0.6666666666666666
Number of requests: 1657
Number of regions: 2
Number of regions in transition: 0
ERROR: hbase:meta is not found on any region.
ERROR: hbase:meta table is not consistent. Run HBCK with proper fix options to fix hbase:meta inconsistency. Exiting...
ERROR: Table lock acquire attempt found:[tableName=
defaultfoo, lockOwner=localhost,60000,1, threadId=2364, purpose=testCheckTableLocks, isShared=false, createTime=63]
Summary:
6 inconsistencies detected.
Status: INCONSISTENT
Standard Error2014-03-03 04:11:20,948 INFO [pool-1-thread-1] hbase.ResourceChecker(147): before: util.TestHBaseFsck#testCheckTableLocks Thread=496, OpenFileDescriptor=821, MaxFileDescriptor=65000, SystemLoadAverage=438, ProcessCount=124, AvailableMemoryMB=2071, ConnectionCount=5
2014-03-03 04:11:20,977 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): master:55261-0x14487da19060000, quorum=localhost:58610, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/balancer
2014-03-03 04:11:20,978 INFO [FifoRpcScheduler.handler1-thread-2] master.HMaster(1538): Client=jenkins//172.18.145.66 set balanceSwitch=false
2014-03-03 04:11:20,996 INFO [Thread-1210] zookeeper.RecoverableZooKeeper(120): Process identifier=hbase Fsck connecting to ZooKeeper ensemble=localhost:58610
2014-03-03 04:11:20,997 DEBUG [Thread-1210-EventThread] zookeeper.ZooKeeperWatcher(310): hbase Fsck, quorum=localhost:58610, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null
2014-03-03 04:11:20,998 DEBUG [Thread-1210-EventThread] zookeeper.ZooKeeperWatcher(387): hbase Fsck-0x14487da190600ad connected
2014-03-03 04:11:21,002 INFO [Thread-1210] util.HBaseFsck(437): Loading regionsinfo from the hbase:meta table
2014-03-03 04:11:21,007 INFO [Thread-1210] util.HBaseFsck(2725): getHTableDescriptors == tableNames => [hbase:namespace]
2014-03-03 04:11:21,014 DEBUG [Thread-1210] util.HBaseFsck(1453): Loading region dirs from hdfs://localhost:47748/user/jenkins/hbase/data/hbase/meta
2014-03-03 04:11:21,014 DEBUG [Thread-1210] util.HBaseFsck(1453): Loading region dirs from hdfs://localhost:47748/user/jenkins/hbase/data/hbase/namespace
2014-03-03 04:11:21,056 INFO [Thread-1210] zookeeper.RecoverableZooKeeper(120): Process identifier=hbase Fsck connecting to ZooKeeper ensemble=localhost:58610
2014-03-03 04:11:21,060 DEBUG [Thread-1210-EventThread] zookeeper.ZooKeeperWatcher(310): hbase Fsck, quorum=localhost:58610, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null
2014-03-03 04:11:21,062 DEBUG [Thread-1210-EventThread] zookeeper.ZooKeeperWatcher(387): hbase Fsck-0x14487da190600ae connected
2014-03-03 04:11:21,069 DEBUG [Thread-1210] util.HBaseFsck(1931): There are 2 region info entries
2014-03-03 04:11:21,076 INFO [Thread-1210] util.HBaseFsck$TableInfo(2575): Handling overlap merges in parallel. set hbasefsck.overlap.merge.parallel to false to run serially.
2014-03-03 04:11:21,077 INFO [Thread-1210] util.HBaseFsck$TableInfo(2575): Handling overlap merges in parallel. set hbasefsck.overlap.merge.parallel to false to run serially.
2014-03-03 04:11:21,102 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): master:55261-0x14487da19060000, quorum=localhost:58610, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/balancer
2014-03-03 04:11:21,104 INFO [FifoRpcScheduler.handler1-thread-2] master.HMaster(1538): Client=jenkins//172.18.145.66 set balanceSwitch=true
2014-03-03 04:11:21,132 INFO [Thread-1210] zookeeper.RecoverableZooKeeper(120): Process identifier=hbase Fsck connecting to ZooKeeper ensemble=localhost:58610
2014-03-03 04:11:21,165 DEBUG [Thread-1210-EventThread] zookeeper.ZooKeeperWatcher(310): hbase Fsck, quorum=localhost:58610, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null
2014-03-03 04:11:21,167 DEBUG [Thread-1210-EventThread] zookeeper.ZooKeeperWatcher(387): hbase Fsck-0x14487da190600af connected
2014-03-03 04:11:21,178 INFO [Thread-1210] zookeeper.RecoverableZooKeeper(120): Process identifier=testing utility connecting to ZooKeeper ensemble=localhost:58610
2014-03-03 04:11:21,194 DEBUG [Thread-1210-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility, quorum=localhost:58610, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null
2014-03-03 04:11:21,195 DEBUG [Thread-1210-EventThread] zookeeper.ZooKeeperWatcher(387): testing utility-0x14487da190600b0 connected
2014-03-03 04:11:21,220 DEBUG [Thread-1210] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/foo/write-testing utility0000000000
2014-03-03 04:11:21,228 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): master:55261-0x14487da19060000, quorum=localhost:58610, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/balancer
2014-03-03 04:11:21,229 INFO [FifoRpcScheduler.handler1-thread-2] master.HMaster(1538): Client=jenkins//172.18.145.66 set balanceSwitch=false
2014-03-03 04:11:21,240 INFO [Thread-1210] zookeeper.RecoverableZooKeeper(120): Process identifier=hbase Fsck connecting to ZooKeeper ensemble=localhost:58610
2014-03-03 04:11:21,249 DEBUG [Thread-1210-EventThread] zookeeper.ZooKeeperWatcher(310): hbase Fsck, quorum=localhost:58610, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null
2014-03-03 04:11:21,251 DEBUG [Thread-1210-EventThread] zookeeper.ZooKeeperWatcher(387): hbase Fsck-0x14487da190600b1 connected
2014-03-03 04:11:21,261 INFO [Thread-1210] util.HBaseFsck(437): Loading regionsinfo from the hbase:meta table
2014-03-03 04:11:21,273 INFO [Thread-1210] util.HBaseFsck(2725): getHTableDescriptors == tableNames => [hbase:namespace]
2014-03-03 04:11:21,280 DEBUG [Thread-1210] util.HBaseFsck(1453): Loading region dirs from hdfs://localhost:47748/user/jenkins/hbase/data/hbase/meta
2014-03-03 04:11:21,280 DEBUG [Thread-1210] util.HBaseFsck(1453): Loading region dirs from hdfs://localhost:47748/user/jenkins/hbase/data/hbase/namespace
2014-03-03 04:11:21,314 INFO [Thread-1210] zookeeper.RecoverableZooKeeper(120): Process identifier=hbase Fsck connecting to ZooKeeper ensemble=localhost:58610
2014-03-03 04:11:21,320 DEBUG [Thread-1210-EventThread] zookeeper.ZooKeeperWatcher(310): hbase Fsck, quorum=localhost:58610, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null
2014-03-03 04:11:21,321 DEBUG [Thread-1210-EventThread] zookeeper.ZooKeeperWatcher(387): hbase Fsck-0x14487da190600b2 connected
2014-03-03 04:11:21,327 DEBUG [Thread-1210] util.HBaseFsck(1931): There are 2 region info entries
2014-03-03 04:11:21,337 INFO [Thread-1210] util.HBaseFsck$TableInfo(2575): Handling overlap merges in parallel. set hbasefsck.overlap.merge.parallel to false to run serially.
2014-03-03 04:11:21,337 INFO [Thread-1210] util.HBaseFsck$TableInfo(2575): Handling overlap merges in parallel. set hbasefsck.overlap.merge.parallel to false to run serially.
2014-03-03 04:11:21,344 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): master:55261-0x14487da19060000, quorum=localhost:58610, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/balancer
2014-03-03 04:11:21,345 INFO [FifoRpcScheduler.handler1-thread-2] master.HMaster(1538): Client=jenkins//172.18.145.66 set balanceSwitch=true
2014-03-03 04:11:21,382 INFO [Thread-1210] zookeeper.RecoverableZooKeeper(120): Process identifier=hbase Fsck connecting to ZooKeeper ensemble=localhost:58610
2014-03-03 04:11:21,394 DEBUG [Thread-1210-EventThread] zookeeper.ZooKeeperWatcher(310): hbase Fsck, quorum=localhost:58610, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null
2014-03-03 04:11:21,395 DEBUG [Thread-1210-EventThread] zookeeper.ZooKeeperWatcher(387): hbase Fsck-0x14487da190600b3 connected
2014-03-03 04:11:21,396 INFO [Thread-1210] lock.ZKInterProcessLockBase(436): Lock is held by: write-testing utility0000000000
2014-03-03 04:11:21,400 WARN [Thread-1210] client.ConnectionManager$HConnectionImplementation(1998): Checking master connection
com.google.protobuf.ServiceException: java.io.IOException: Call to c64-s15.cs1cloud.internal/172.18.145.66:55261 failed on local exception: org.apache.hadoop.hbase.ipc.RpcClient$CallTimeoutException: Call id=1360, waitTime=0, rpcTimeout=0
at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1699)
at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1740)
at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:40216)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceState.isMasterRunning(ConnectionManager.java:1358)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.isKeepAliveMasterConnectedAndRunning(ConnectionManager.java:1991)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1710)
at org.apache.hadoop.hbase.client.HBaseAdmin$MasterCallable.prepare(HBaseAdmin.java:3199)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:120)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:97)
at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3226)
at org.apache.hadoop.hbase.client.HBaseAdmin.getClusterStatus(HBaseAdmin.java:2158)
at org.apache.hadoop.hbase.util.HBaseFsck.connect(HBaseFsck.java:308)
at org.apache.hadoop.hbase.util.hbck.HbckTestingUtil.doFsck(HbckTestingUtil.java:52)
at org.apache.hadoop.hbase.util.hbck.HbckTestingUtil.doFsck(HbckTestingUtil.java:43)
at org.apache.hadoop.hbase.util.hbck.HbckTestingUtil.doFsck(HbckTestingUtil.java:38)
at org.apache.hadoop.hbase.util.TestHBaseFsck.testCheckTableLocks(TestHBaseFsck.java:2100)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
Caused by: java.io.IOException: Call to c64-s15.cs1cloud.internal/172.18.145.66:55261 failed on local exception: org.apache.hadoop.hbase.ipc.RpcClient$CallTimeoutException: Call id=1360, waitTime=0, rpcTimeout=0
at org.apache.hadoop.hbase.ipc.RpcClient.wrapException(RpcClient.java:1516)
at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1486)
at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1684)
... 24 more
Caused by: org.apache.hadoop.hbase.ipc.RpcClient$CallTimeoutException: Call id=1360, waitTime=0, rpcTimeout=0
at org.apache.hadoop.hbase.ipc.RpcClient$Connection.cleanupCalls(RpcClient.java:1214)
at org.apache.hadoop.hbase.ipc.RpcClient$Connection.cleanupCalls(RpcClient.java:1205)
at org.apache.hadoop.hbase.ipc.RpcClient$Connection.close(RpcClient.java:1006)
at org.apache.hadoop.hbase.ipc.RpcClient$Connection.run(RpcClient.java:749)
2014-03-03 04:11:21,419 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): master:55261-0x14487da19060000, quorum=localhost:58610, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/balancer
2014-03-03 04:11:21,420 INFO [FifoRpcScheduler.handler1-thread-2] master.HMaster(1538): Client=jenkins//172.18.145.66 set balanceSwitch=false
2014-03-03 04:11:21,424 WARN [Thread-1210] util.HBaseFsck(1551): Could not process regionserver c64-s15.cs1cloud.internal:52297
java.io.IOException: Call to c64-s15.cs1cloud.internal/172.18.145.66:52297 failed on local exception: org.apache.hadoop.hbase.ipc.RpcClient$CallTimeoutException: Call id=1364, waitTime=0, rpcTimeout=0
at org.apache.hadoop.hbase.ipc.RpcClient.wrapException(RpcClient.java:1516)
at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1486)
at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1684)
at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1740)
at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.getOnlineRegion(AdminProtos.java:20583)
at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getOnlineRegions(ProtobufUtil.java:1693)
at org.apache.hadoop.hbase.util.HBaseFsck$WorkItemRegion.call(HBaseFsck.java:3377)
at org.apache.hadoop.hbase.util.HBaseFsck$WorkItemRegion.call(HBaseFsck.java:3356)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.hadoop.hbase.ipc.RpcClient$CallTimeoutException: Call id=1364, waitTime=0, rpcTimeout=0
at org.apache.hadoop.hbase.ipc.RpcClient$Connection.cleanupCalls(RpcClient.java:1214)
at org.apache.hadoop.hbase.ipc.RpcClient$Connection.cleanupCalls(RpcClient.java:1205)
at org.apache.hadoop.hbase.ipc.RpcClient$Connection.close(RpcClient.java:1006)
at org.apache.hadoop.hbase.ipc.RpcClient$Connection.run(RpcClient.java:749)
2014-03-03 04:11:21,424 WARN [Thread-1210] util.HBaseFsck(1551): Could not process regionserver c64-s15.cs1cloud.internal:53849
java.io.IOException: Call to c64-s15.cs1cloud.internal/172.18.145.66:53849 failed on local exception: org.apache.hadoop.hbase.ipc.RpcClient$CallTimeoutException: Call id=1365, waitTime=0, rpcTimeout=0
at org.apache.hadoop.hbase.ipc.RpcClient.wrapException(RpcClient.java:1516)
at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1486)
at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1684)
at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1740)
at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.getOnlineRegion(AdminProtos.java:20583)
at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getOnlineRegions(ProtobufUtil.java:1693)
at org.apache.hadoop.hbase.util.HBaseFsck$WorkItemRegion.call(HBaseFsck.java:3377)
at org.apache.hadoop.hbase.util.HBaseFsck$WorkItemRegion.call(HBaseFsck.java:3356)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.hadoop.hbase.ipc.RpcClient$CallTimeoutException: Call id=1365, waitTime=0, rpcTimeout=0
at org.apache.hadoop.hbase.ipc.RpcClient$Connection.cleanupCalls(RpcClient.java:1214)
at org.apache.hadoop.hbase.ipc.RpcClient$Connection.cleanupCalls(RpcClient.java:1205)
at org.apache.hadoop.hbase.ipc.RpcClient$Connection.close(RpcClient.java:1006)
at org.apache.hadoop.hbase.ipc.RpcClient$Connection.run(RpcClient.java:749)
2014-03-03 04:11:21,425 WARN [Thread-1210] util.HBaseFsck(1551): Could not process regionserver c64-s15.cs1cloud.internal:38669
java.io.IOException: Call to c64-s15.cs1cloud.internal/172.18.145.66:38669 failed on local exception: org.apache.hadoop.hbase.ipc.RpcClient$CallTimeoutException: Call id=1366, waitTime=0, rpcTimeout=0
at org.apache.hadoop.hbase.ipc.RpcClient.wrapException(RpcClient.java:1516)
at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1486)
at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1684)
at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1740)
at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.getOnlineRegion(AdminProtos.java:20583)
at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getOnlineRegions(ProtobufUtil.java:1693)
at org.apache.hadoop.hbase.util.HBaseFsck$WorkItemRegion.call(HBaseFsck.java:3377)
at org.apache.hadoop.hbase.util.HBaseFsck$WorkItemRegion.call(HBaseFsck.java:3356)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.hadoop.hbase.ipc.RpcClient$CallTimeoutException: Call id=1366, waitTime=0, rpcTimeout=0
at org.apache.hadoop.hbase.ipc.RpcClient$Connection.cleanupCalls(RpcClient.java:1214)
at org.apache.hadoop.hbase.ipc.RpcClient$Connection.cleanupCalls(RpcClient.java:1205)
at org.apache.hadoop.hbase.ipc.RpcClient$Connection.close(RpcClient.java:1006)
at org.apache.hadoop.hbase.ipc.RpcClient$Connection.run(RpcClient.java:749)
2014-03-03 04:11:21,436 INFO [Thread-1210] zookeeper.RecoverableZooKeeper(120): Process identifier=hbase Fsck connecting to ZooKeeper ensemble=localhost:58610
2014-03-03 04:11:21,444 DEBUG [Thread-1210-EventThread] zookeeper.ZooKeeperWatcher(310): hbase Fsck, quorum=localhost:58610, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null
2014-03-03 04:11:21,445 DEBUG [Thread-1210-EventThread] zookeeper.ZooKeeperWatcher(387): hbase Fsck-0x14487da190600b4 connected
2014-03-03 04:11:21,460 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): master:55261-0x14487da19060000, quorum=localhost:58610, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/balancer
2014-03-03 04:11:21,462 INFO [FifoRpcScheduler.handler1-thread-2] master.HMaster(1538): Client=jenkins//172.18.145.66 set balanceSwitch=true
2014-03-03 04:11:21,490 INFO [Thread-1210] zookeeper.RecoverableZooKeeper(120): Process identifier=hbase Fsck connecting to ZooKeeper ensemble=localhost:58610
2014-03-03 04:11:21,494 DEBUG [Thread-1210-EventThread] zookeeper.ZooKeeperWatcher(310): hbase Fsck, quorum=localhost:58610, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null
2014-03-03 04:11:21,495 DEBUG [Thread-1210-EventThread] zookeeper.ZooKeeperWatcher(387): hbase Fsck-0x14487da190600b5 connected
2014-03-03 04:11:21,496 INFO [Thread-1210] lock.ZKInterProcessLockBase(436): Lock is held by: write-testing utility0000000000
2014-03-03 04:11:21,538 INFO [pool-1-thread-1] hbase.ResourceChecker(171): after: util.TestHBaseFsck#testCheckTableLocks Thread=500 (was 496)
Potentially hanging thread: Thread-1210-SendThread(localhost:58610)
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
Potentially hanging thread: Thread-1210-SendThread(localhost:58610)
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1821155657_8 at /127.0.0.1:40769 [Waiting for operation #14]
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
java.io.BufferedInputStream.read(BufferedInputStream.java:237)
java.io.DataInputStream.readShort(DataInputStream.java:295)
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:52)
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:198)
java.lang.Thread.run(Thread.java:662)
Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1821155657_8 at /127.0.0.1:38011 [Waiting for operation #7]
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
java.io.BufferedInputStream.read(BufferedInputStream.java:237)
java.io.DataInputStream.readShort(DataInputStream.java:295)
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:52)
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:198)
java.lang.Thread.run(Thread.java:662)
Potentially hanging thread: Thread-1210-SendThread(localhost:58610)
java.lang.Thread.sleep(Native Method)
org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:219)
org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1157)
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1109)
Potentially hanging thread: Thread-1210-EventThread
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
Potentially hanging thread: Thread-1210-EventThread
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
Potentially hanging thread: Thread-1210-EventThread
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
Potentially hanging thread: Thread-1210-SendThread(localhost:58610)
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
Potentially hanging thread: Thread-1210-EventThread
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
Potentially hanging thread: Thread-1210-SendThread(localhost:58610)
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1821155657_8 at /127.0.0.1:35231 [Waiting for operation #17]
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
java.io.BufferedInputStream.read(BufferedInputStream.java:237)
java.io.DataInputStream.readShort(DataInputStream.java:295)
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:52)
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:198)
java.lang.Thread.run(Thread.java:662)
- Thread LEAK? -, OpenFileDescriptor=837 (was 821) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=65000 (was 65000), SystemLoadAverage=438 (was 438), ProcessCount=124 (was 124), AvailableMemoryMB=2054 (was 2071), ConnectionCount=5 (was 5)
2014-03-03 04:15:02,092 DEBUG [Thread-1210-EventThread] zookeeper.ZooKeeperWatcher(310): hbase Fsck-0x14487da190600af, quorum=localhost:58610, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Disconnected, path=null
2014-03-03 04:15:02,092 DEBUG [Thread-1210-EventThread] zookeeper.ZooKeeperWatcher(310): hbase Fsck-0x14487da190600b5, quorum=localhost:58610, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Disconnected, path=null
2014-03-03 04:15:02,092 DEBUG [Thread-1210-EventThread] zookeeper.ZooKeeperWatcher(392): hbase Fsck-0x14487da190600af, quorum=localhost:58610, baseZNode=/hbase Received Disconnected from ZooKeeper, ignoring
2014-03-03 04:15:02,092 DEBUG [Thread-1210-EventThread] zookeeper.ZooKeeperWatcher(392): hbase Fsck-0x14487da190600b5, quorum=localhost:58610, baseZNode=/hbase Received Disconnected from ZooKeeper, ignoring
2014-03-03 04:15:02,101 DEBUG [Thread-1210-EventThread] zookeeper.ZooKeeperWatcher(310): hbase Fsck-0x14487da190600b3, quorum=localhost:58610, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Disconnected, path=null
2014-03-03 04:15:02,101 DEBUG [Thread-1210-EventThread] zookeeper.ZooKeeperWatcher(392): hbase Fsck-0x14487da190600b3, quorum=localhost:58610, baseZNode=/hbase Received Disconnected from ZooKeeper, ignoring
|