Regression org.apache.hadoop.hbase.procedure.TestZKProcedure.testMultiCohortWithMemberTimeoutDuringPrepare Failing for the past 1 build (Since Failed#364 ) Took 16 ms. Error Message subprocedureImpl.acquireBarrier(); Wanted 1 time: -> at org.apache.hadoop.hbase.procedure.TestZKProcedure.waitAndVerifySubproc(TestZKProcedure.java:371) But was 2 times. Undesired invocation: -> at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:161) Stacktrace org.mockito.exceptions.verification.TooManyActualInvocations: subprocedureImpl.acquireBarrier(); Wanted 1 time: -> at org.apache.hadoop.hbase.procedure.TestZKProcedure.waitAndVerifySubproc(TestZKProcedure.java:371) But was 2 times. Undesired invocation: -> at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:161) at org.apache.hadoop.hbase.procedure.TestZKProcedure.waitAndVerifySubproc(TestZKProcedure.java:371) at org.apache.hadoop.hbase.procedure.TestZKProcedure.verifyCohortSuccessful(TestZKProcedure.java:393) at org.apache.hadoop.hbase.procedure.TestZKProcedure.testMultiCohortWithMemberTimeoutDuringPrepare(TestZKProcedure.java:325) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.run(ParentRunner.java:309) at org.junit.runners.Suite.runChild(Suite.java:127) at org.junit.runners.Suite.runChild(Suite.java:26) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) Standard Output 2014-06-29 19:26:30,087 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(389): Created new mini-cluster data directory: /home/jenkins/jenkins-slave/workspace/HBase-0.98/hbase-server/target/test-data/a49d83df-5426-4597-817c-42f1ce4a21d9/dfscluster_4f680485-654b-4ab1-9fc5-ec512da39b71, deleteOnExit=true 2014-06-29 19:26:30,266 INFO [pool-1-thread-1] zookeeper.MiniZooKeeperCluster(200): Started MiniZK Cluster and connect 1 ZK server on client port: 51439 2014-06-29 19:26:30,337 INFO [pool-1-thread-1] hbase.ResourceChecker(147): before: procedure.TestZKProcedure#testSingleMember Thread=9, OpenFileDescriptor=130, MaxFileDescriptor=40000, SystemLoadAverage=374, ProcessCount=168, AvailableMemoryMB=3166, ConnectionCount=0 2014-06-29 19:26:30,384 INFO [pool-1-thread-1] zookeeper.RecoverableZooKeeper(120): Process identifier=testing utility connecting to ZooKeeper ensemble=localhost:51439 2014-06-29 19:26:31,050 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2014-06-29 19:26:31,055 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(387): testing utility-0x146e917cf2b0000 connected 2014-06-29 19:26:31,169 INFO [pool-1-thread-1] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/coordination test - 1 cohort members/acquired /hbase/coordination test - 1 cohort members/reached /hbase/coordination test - 1 cohort members/abort 2014-06-29 19:26:31,173 DEBUG [pool-1-thread-1] procedure.ZKProcedureCoordinatorRpcs(195): Starting the controller for procedure member:coordinator 2014-06-29 19:26:31,295 INFO [pool-1-thread-1] zookeeper.RecoverableZooKeeper(120): Process identifier=testing utility connecting to ZooKeeper ensemble=localhost:51439 2014-06-29 19:26:31,715 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2014-06-29 19:26:31,716 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(387): testing utility-0x146e917cf2b0001 connected 2014-06-29 19:26:31,821 INFO [pool-1-thread-1] zookeeper.RecoverableZooKeeper(508): Node /hbase/coordination test - 1 cohort members/acquired already exists and this is not a retry 2014-06-29 19:26:31,825 DEBUG [pool-1-thread-1] procedure.ZKProcedureMemberRpcs(337): Starting procedure member 'one' 2014-06-29 19:26:31,825 DEBUG [pool-1-thread-1] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/coordination test - 1 cohort members/abort' 2014-06-29 19:26:31,826 DEBUG [pool-1-thread-1] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/coordination test - 1 cohort members/acquired' 2014-06-29 19:26:31,895 INFO [(coordinator)-proc-coordinator-pool1-thread-1] procedure.Procedure(193): Starting procedure 'op' 2014-06-29 19:26:31,897 DEBUG [(coordinator)-proc-coordinator-pool1-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 60000 ms 2014-06-29 19:26:31,898 DEBUG [(coordinator)-proc-coordinator-pool1-thread-1] procedure.Procedure(201): Procedure 'op' starting 'acquire' 2014-06-29 19:26:31,899 DEBUG [(coordinator)-proc-coordinator-pool1-thread-1] procedure.Procedure(243): Starting procedure 'op', kicking off acquire phase on members. 2014-06-29 19:26:31,906 DEBUG [(coordinator)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(430): testing utility-0x146e917cf2b0000, quorum=localhost:51439, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/coordination test - 1 cohort members/abort/op 2014-06-29 19:26:31,906 DEBUG [(coordinator)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(93): Creating acquire znode:/hbase/coordination test - 1 cohort members/acquired/op 2014-06-29 19:26:32,066 WARN [(coordinator)-proc-coordinator-pool1-thread-1] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2014-06-29 19:26:32,260 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility-0x146e917cf2b0001, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/coordination test - 1 cohort members/acquired 2014-06-29 19:26:32,260 DEBUG [(coordinator)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(101): Watching for acquire node:/hbase/coordination test - 1 cohort members/acquired/op/one 2014-06-29 19:26:32,260 INFO [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs$1(106): Received procedure start children changed event: /hbase/coordination test - 1 cohort members/acquired 2014-06-29 19:26:32,260 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/coordination test - 1 cohort members/acquired' 2014-06-29 19:26:32,261 DEBUG [(coordinator)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(430): testing utility-0x146e917cf2b0000, quorum=localhost:51439, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/coordination test - 1 cohort members/acquired/op/one 2014-06-29 19:26:32,261 DEBUG [(coordinator)-proc-coordinator-pool1-thread-1] procedure.Procedure(205): Waiting for all members to 'acquire' 2014-06-29 19:26:32,262 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(184): Found procedure znode: /hbase/coordination test - 1 cohort members/acquired/op 2014-06-29 19:26:32,262 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(430): testing utility-0x146e917cf2b0001, quorum=localhost:51439, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/coordination test - 1 cohort members/abort/op 2014-06-29 19:26:32,265 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(203): start proc data length is 6 2014-06-29 19:26:32,265 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(212): Found data for znode:/hbase/coordination test - 1 cohort members/acquired/op 2014-06-29 19:26:32,267 DEBUG [pool-1-thread-1-EventThread] procedure.TestZKProcedure$3(173): Task size:1, getting:0 2014-06-29 19:26:32,282 DEBUG [pool-1-thread-1-EventThread] procedure.ProcedureMember(155): Submitting new Subprocedure:op 2014-06-29 19:26:32,283 DEBUG [member: 'one' subprocedure-pool2-thread-1] procedure.Subprocedure(152): Starting subprocedure 'op' with timeout 10000ms 2014-06-29 19:26:32,283 DEBUG [member: 'one' subprocedure-pool2-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 10000 ms 2014-06-29 19:26:32,283 DEBUG [member: 'one' subprocedure-pool2-thread-1] procedure.Subprocedure(160): Subprocedure 'op' starting 'acquire' stage 2014-06-29 19:26:32,288 DEBUG [member: 'one' subprocedure-pool2-thread-1] procedure.Subprocedure(162): Subprocedure 'op' locally acquired 2014-06-29 19:26:32,288 DEBUG [member: 'one' subprocedure-pool2-thread-1] procedure.ZKProcedureMemberRpcs(237): Member: 'one' joining acquired barrier for procedure (op) in zk 2014-06-29 19:26:32,301 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility-0x146e917cf2b0000, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/coordination test - 1 cohort members/acquired/op/one 2014-06-29 19:26:32,301 DEBUG [member: 'one' subprocedure-pool2-thread-1] procedure.ZKProcedureMemberRpcs(245): Watch for global barrier reached:/hbase/coordination test - 1 cohort members/reached/op 2014-06-29 19:26:32,302 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(170): Node created: /hbase/coordination test - 1 cohort members/acquired/op/one 2014-06-29 19:26:32,302 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2014-06-29 19:26:32,302 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/coordination test - 1 cohort members 2014-06-29 19:26:32,302 DEBUG [member: 'one' subprocedure-pool2-thread-1] zookeeper.ZKUtil(430): testing utility-0x146e917cf2b0001, quorum=localhost:51439, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/coordination test - 1 cohort members/reached/op 2014-06-29 19:26:32,303 DEBUG [member: 'one' subprocedure-pool2-thread-1] procedure.Subprocedure(166): Subprocedure 'op' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2014-06-29 19:26:32,303 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2014-06-29 19:26:32,304 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |----op 2014-06-29 19:26:32,304 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-------one 2014-06-29 19:26:32,305 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-abort 2014-06-29 19:26:32,306 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-reached 2014-06-29 19:26:32,307 DEBUG [pool-1-thread-1-EventThread] procedure.Procedure(294): member: 'one' joining acquired barrier for procedure 'op' on coordinator 2014-06-29 19:26:32,307 DEBUG [(coordinator)-proc-coordinator-pool1-thread-1] procedure.Procedure(209): Procedure 'op' starting 'in-barrier' execution. 2014-06-29 19:26:32,307 DEBUG [pool-1-thread-1-EventThread] procedure.Procedure(303): Waiting on: java.util.concurrent.CountDownLatch@e5f46e[Count = 0] remaining members to acquire global barrier 2014-06-29 19:26:32,308 DEBUG [(coordinator)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(115): Creating reached barrier zk node:/hbase/coordination test - 1 cohort members/reached/op 2014-06-29 19:26:32,322 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility-0x146e917cf2b0001, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/coordination test - 1 cohort members/reached/op 2014-06-29 19:26:32,323 INFO [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs$1(79): Received created event:/hbase/coordination test - 1 cohort members/reached/op 2014-06-29 19:26:32,323 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(130): Recieved reached global barrier:/hbase/coordination test - 1 cohort members/reached/op 2014-06-29 19:26:32,323 DEBUG [(coordinator)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(430): testing utility-0x146e917cf2b0000, quorum=localhost:51439, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/coordination test - 1 cohort members/reached/op/one 2014-06-29 19:26:32,323 DEBUG [member: 'one' subprocedure-pool2-thread-1] procedure.Subprocedure(180): Subprocedure 'op' received 'reached' from coordinator. 2014-06-29 19:26:32,324 DEBUG [member: 'one' subprocedure-pool2-thread-1] procedure.Subprocedure(182): Subprocedure 'op' locally completed 2014-06-29 19:26:32,324 DEBUG [member: 'one' subprocedure-pool2-thread-1] procedure.ZKProcedureMemberRpcs(261): Marking procedure 'op' completed for member 'one' in zk 2014-06-29 19:26:32,334 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility-0x146e917cf2b0000, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/coordination test - 1 cohort members/reached/op/one 2014-06-29 19:26:32,335 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(170): Node created: /hbase/coordination test - 1 cohort members/reached/op/one 2014-06-29 19:26:32,335 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2014-06-29 19:26:32,335 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/coordination test - 1 cohort members 2014-06-29 19:26:32,335 DEBUG [member: 'one' subprocedure-pool2-thread-1] procedure.Subprocedure(186): Subprocedure 'op' has notified controller of completion 2014-06-29 19:26:32,335 DEBUG [member: 'one' subprocedure-pool2-thread-1] errorhandling.TimeoutExceptionInjector(89): Marking timer as complete - no error notifications will be received for this timer. 2014-06-29 19:26:32,336 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2014-06-29 19:26:32,336 DEBUG [member: 'one' subprocedure-pool2-thread-1] procedure.Subprocedure(211): Subprocedure 'op' completed. 2014-06-29 19:26:32,337 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |----op 2014-06-29 19:26:32,337 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-------one 2014-06-29 19:26:32,338 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-abort 2014-06-29 19:26:32,339 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-reached 2014-06-29 19:26:32,340 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |----op 2014-06-29 19:26:32,340 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-------one 2014-06-29 19:26:32,341 INFO [(coordinator)-proc-coordinator-pool1-thread-1] procedure.Procedure(217): Procedure 'op' execution completed 2014-06-29 19:26:32,341 DEBUG [pool-1-thread-1-EventThread] procedure.Procedure(324): Member: 'one' released barrier for procedure'op', counting down latch. Waiting for 0 more 2014-06-29 19:26:32,341 DEBUG [(coordinator)-proc-coordinator-pool1-thread-1] procedure.Procedure(226): Running finish phase. 2014-06-29 19:26:32,342 DEBUG [(coordinator)-proc-coordinator-pool1-thread-1] procedure.Procedure(277): Finished coordinator procedure - removing self from list of running procedures 2014-06-29 19:26:32,342 DEBUG [(coordinator)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(141): Attempting to clean out zk node for op:op 2014-06-29 19:26:32,342 INFO [(coordinator)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureUtil(284): Clearing all znodes for procedure opincluding nodes /hbase/coordination test - 1 cohort members/acquired /hbase/coordination test - 1 cohort members/reached /hbase/coordination test - 1 cohort members/abort 2014-06-29 19:26:32,372 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility-0x146e917cf2b0001, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/coordination test - 1 cohort members/acquired 2014-06-29 19:26:32,373 INFO [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs$1(106): Received procedure start children changed event: /hbase/coordination test - 1 cohort members/acquired 2014-06-29 19:26:32,373 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/coordination test - 1 cohort members/acquired' 2014-06-29 19:26:32,406 DEBUG [(coordinator)-proc-coordinator-pool1-thread-1] errorhandling.TimeoutExceptionInjector(89): Marking timer as complete - no error notifications will be received for this timer. 2014-06-29 19:26:32,415 DEBUG [pool-1-thread-1] procedure.TestZKProcedure(392): Checking mock:0 2014-06-29 19:26:32,435 INFO [pool-1-thread-1] hbase.ResourceChecker(171): after: procedure.TestZKProcedure#testSingleMember Thread=13 (was 9) Potentially hanging thread: pool-1-thread-1-SendThread(localhost:51439) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) Potentially hanging thread: pool-1-thread-1-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) Potentially hanging thread: pool-1-thread-1-SendThread(localhost:51439) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) Potentially hanging thread: pool-1-thread-1-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) - Thread LEAK? -, OpenFileDescriptor=142 (was 130) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=40000 (was 40000), SystemLoadAverage=374 (was 374), ProcessCount=168 (was 168), AvailableMemoryMB=3169 (was 3166) - AvailableMemoryMB LEAK? -, ConnectionCount=0 (was 0) 2014-06-29 19:26:32,449 INFO [pool-1-thread-1] hbase.ResourceChecker(147): before: procedure.TestZKProcedure#testMultipleMembers Thread=13, OpenFileDescriptor=144, MaxFileDescriptor=40000, SystemLoadAverage=374, ProcessCount=168, AvailableMemoryMB=3169, ConnectionCount=0 2014-06-29 19:26:32,481 INFO [pool-1-thread-1] zookeeper.RecoverableZooKeeper(120): Process identifier=testing utility connecting to ZooKeeper ensemble=localhost:51439 2014-06-29 19:26:32,500 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2014-06-29 19:26:32,501 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(387): testing utility-0x146e917cf2b0002 connected 2014-06-29 19:26:32,564 INFO [pool-1-thread-1] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/coordination test - 4 cohort members/acquired /hbase/coordination test - 4 cohort members/reached /hbase/coordination test - 4 cohort members/abort 2014-06-29 19:26:32,566 DEBUG [pool-1-thread-1] procedure.ZKProcedureCoordinatorRpcs(195): Starting the controller for procedure member:coordinator 2014-06-29 19:26:32,569 INFO [pool-1-thread-1] zookeeper.RecoverableZooKeeper(120): Process identifier=testing utility connecting to ZooKeeper ensemble=localhost:51439 2014-06-29 19:26:32,581 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2014-06-29 19:26:32,582 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(387): testing utility-0x146e917cf2b0003 connected 2014-06-29 19:26:32,597 INFO [pool-1-thread-1] zookeeper.RecoverableZooKeeper(508): Node /hbase/coordination test - 4 cohort members/acquired already exists and this is not a retry 2014-06-29 19:26:32,599 DEBUG [pool-1-thread-1] procedure.ZKProcedureMemberRpcs(337): Starting procedure member 'one' 2014-06-29 19:26:32,599 DEBUG [pool-1-thread-1] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/coordination test - 4 cohort members/abort' 2014-06-29 19:26:32,600 DEBUG [pool-1-thread-1] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/coordination test - 4 cohort members/acquired' 2014-06-29 19:26:32,603 INFO [pool-1-thread-1] zookeeper.RecoverableZooKeeper(120): Process identifier=testing utility connecting to ZooKeeper ensemble=localhost:51439 2014-06-29 19:26:32,612 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2014-06-29 19:26:32,613 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(387): testing utility-0x146e917cf2b0004 connected 2014-06-29 19:26:32,623 INFO [pool-1-thread-1] zookeeper.RecoverableZooKeeper(508): Node /hbase/coordination test - 4 cohort members/acquired already exists and this is not a retry 2014-06-29 19:26:32,624 DEBUG [pool-1-thread-1] procedure.ZKProcedureMemberRpcs(337): Starting procedure member 'two' 2014-06-29 19:26:32,624 DEBUG [pool-1-thread-1] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/coordination test - 4 cohort members/abort' 2014-06-29 19:26:32,625 DEBUG [pool-1-thread-1] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/coordination test - 4 cohort members/acquired' 2014-06-29 19:26:32,628 INFO [pool-1-thread-1] zookeeper.RecoverableZooKeeper(120): Process identifier=testing utility connecting to ZooKeeper ensemble=localhost:51439 2014-06-29 19:26:32,634 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2014-06-29 19:26:32,636 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(387): testing utility-0x146e917cf2b0005 connected 2014-06-29 19:26:32,645 INFO [pool-1-thread-1] zookeeper.RecoverableZooKeeper(508): Node /hbase/coordination test - 4 cohort members/acquired already exists and this is not a retry 2014-06-29 19:26:32,646 DEBUG [pool-1-thread-1] procedure.ZKProcedureMemberRpcs(337): Starting procedure member 'three' 2014-06-29 19:26:32,647 DEBUG [pool-1-thread-1] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/coordination test - 4 cohort members/abort' 2014-06-29 19:26:32,647 DEBUG [pool-1-thread-1] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/coordination test - 4 cohort members/acquired' 2014-06-29 19:26:32,650 INFO [pool-1-thread-1] zookeeper.RecoverableZooKeeper(120): Process identifier=testing utility connecting to ZooKeeper ensemble=localhost:51439 2014-06-29 19:26:32,656 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2014-06-29 19:26:32,658 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(387): testing utility-0x146e917cf2b0006 connected 2014-06-29 19:26:32,672 INFO [pool-1-thread-1] zookeeper.RecoverableZooKeeper(508): Node /hbase/coordination test - 4 cohort members/acquired already exists and this is not a retry 2014-06-29 19:26:32,674 DEBUG [pool-1-thread-1] procedure.ZKProcedureMemberRpcs(337): Starting procedure member 'four' 2014-06-29 19:26:32,674 DEBUG [pool-1-thread-1] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/coordination test - 4 cohort members/abort' 2014-06-29 19:26:32,675 DEBUG [pool-1-thread-1] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/coordination test - 4 cohort members/acquired' 2014-06-29 19:26:32,678 INFO [(coordinator)-proc-coordinator-pool3-thread-1] procedure.Procedure(193): Starting procedure 'op' 2014-06-29 19:26:32,678 DEBUG [(coordinator)-proc-coordinator-pool3-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 60000 ms 2014-06-29 19:26:32,678 DEBUG [(coordinator)-proc-coordinator-pool3-thread-1] procedure.Procedure(201): Procedure 'op' starting 'acquire' 2014-06-29 19:26:32,679 DEBUG [(coordinator)-proc-coordinator-pool3-thread-1] procedure.Procedure(243): Starting procedure 'op', kicking off acquire phase on members. 2014-06-29 19:26:32,680 DEBUG [(coordinator)-proc-coordinator-pool3-thread-1] zookeeper.ZKUtil(430): testing utility-0x146e917cf2b0002, quorum=localhost:51439, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/coordination test - 4 cohort members/abort/op 2014-06-29 19:26:32,680 DEBUG [(coordinator)-proc-coordinator-pool3-thread-1] procedure.ZKProcedureCoordinatorRpcs(93): Creating acquire znode:/hbase/coordination test - 4 cohort members/acquired/op 2014-06-29 19:26:32,701 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility-0x146e917cf2b0005, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/coordination test - 4 cohort members/acquired 2014-06-29 19:26:32,701 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility-0x146e917cf2b0006, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/coordination test - 4 cohort members/acquired 2014-06-29 19:26:32,701 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility-0x146e917cf2b0004, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/coordination test - 4 cohort members/acquired 2014-06-29 19:26:32,701 DEBUG [(coordinator)-proc-coordinator-pool3-thread-1] procedure.ZKProcedureCoordinatorRpcs(101): Watching for acquire node:/hbase/coordination test - 4 cohort members/acquired/op/one 2014-06-29 19:26:32,701 INFO [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs$1(106): Received procedure start children changed event: /hbase/coordination test - 4 cohort members/acquired 2014-06-29 19:26:32,701 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility-0x146e917cf2b0003, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/coordination test - 4 cohort members/acquired 2014-06-29 19:26:32,702 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/coordination test - 4 cohort members/acquired' 2014-06-29 19:26:32,702 INFO [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs$1(106): Received procedure start children changed event: /hbase/coordination test - 4 cohort members/acquired 2014-06-29 19:26:32,702 DEBUG [(coordinator)-proc-coordinator-pool3-thread-1] zookeeper.ZKUtil(430): testing utility-0x146e917cf2b0002, quorum=localhost:51439, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/coordination test - 4 cohort members/acquired/op/one 2014-06-29 19:26:32,701 INFO [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs$1(106): Received procedure start children changed event: /hbase/coordination test - 4 cohort members/acquired 2014-06-29 19:26:32,702 DEBUG [(coordinator)-proc-coordinator-pool3-thread-1] procedure.ZKProcedureCoordinatorRpcs(101): Watching for acquire node:/hbase/coordination test - 4 cohort members/acquired/op/two 2014-06-29 19:26:32,701 INFO [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs$1(106): Received procedure start children changed event: /hbase/coordination test - 4 cohort members/acquired 2014-06-29 19:26:32,703 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/coordination test - 4 cohort members/acquired' 2014-06-29 19:26:32,703 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/coordination test - 4 cohort members/acquired' 2014-06-29 19:26:32,703 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(184): Found procedure znode: /hbase/coordination test - 4 cohort members/acquired/op 2014-06-29 19:26:32,702 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/coordination test - 4 cohort members/acquired' 2014-06-29 19:26:32,703 DEBUG [(coordinator)-proc-coordinator-pool3-thread-1] zookeeper.ZKUtil(430): testing utility-0x146e917cf2b0002, quorum=localhost:51439, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/coordination test - 4 cohort members/acquired/op/two 2014-06-29 19:26:32,704 DEBUG [(coordinator)-proc-coordinator-pool3-thread-1] procedure.ZKProcedureCoordinatorRpcs(101): Watching for acquire node:/hbase/coordination test - 4 cohort members/acquired/op/three 2014-06-29 19:26:32,704 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(184): Found procedure znode: /hbase/coordination test - 4 cohort members/acquired/op 2014-06-29 19:26:32,704 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(184): Found procedure znode: /hbase/coordination test - 4 cohort members/acquired/op 2014-06-29 19:26:32,704 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(430): testing utility-0x146e917cf2b0005, quorum=localhost:51439, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/coordination test - 4 cohort members/abort/op 2014-06-29 19:26:32,705 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(184): Found procedure znode: /hbase/coordination test - 4 cohort members/acquired/op 2014-06-29 19:26:32,705 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(430): testing utility-0x146e917cf2b0006, quorum=localhost:51439, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/coordination test - 4 cohort members/abort/op 2014-06-29 19:26:32,706 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(430): testing utility-0x146e917cf2b0003, quorum=localhost:51439, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/coordination test - 4 cohort members/abort/op 2014-06-29 19:26:32,706 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(203): start proc data length is 6 2014-06-29 19:26:32,705 DEBUG [(coordinator)-proc-coordinator-pool3-thread-1] zookeeper.ZKUtil(430): testing utility-0x146e917cf2b0002, quorum=localhost:51439, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/coordination test - 4 cohort members/acquired/op/three 2014-06-29 19:26:32,707 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(203): start proc data length is 6 2014-06-29 19:26:32,707 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(203): start proc data length is 6 2014-06-29 19:26:32,707 DEBUG [(coordinator)-proc-coordinator-pool3-thread-1] procedure.ZKProcedureCoordinatorRpcs(101): Watching for acquire node:/hbase/coordination test - 4 cohort members/acquired/op/four 2014-06-29 19:26:32,706 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(212): Found data for znode:/hbase/coordination test - 4 cohort members/acquired/op 2014-06-29 19:26:32,707 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(212): Found data for znode:/hbase/coordination test - 4 cohort members/acquired/op 2014-06-29 19:26:32,707 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(212): Found data for znode:/hbase/coordination test - 4 cohort members/acquired/op 2014-06-29 19:26:32,708 DEBUG [(coordinator)-proc-coordinator-pool3-thread-1] zookeeper.ZKUtil(430): testing utility-0x146e917cf2b0002, quorum=localhost:51439, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/coordination test - 4 cohort members/acquired/op/four 2014-06-29 19:26:32,708 DEBUG [pool-1-thread-1-EventThread] procedure.TestZKProcedure$3(173): Task size:4, getting:0 2014-06-29 19:26:32,708 DEBUG [(coordinator)-proc-coordinator-pool3-thread-1] procedure.Procedure(205): Waiting for all members to 'acquire' 2014-06-29 19:26:32,709 DEBUG [pool-1-thread-1-EventThread] procedure.TestZKProcedure$3(173): Task size:4, getting:1 2014-06-29 19:26:32,708 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(430): testing utility-0x146e917cf2b0004, quorum=localhost:51439, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/coordination test - 4 cohort members/abort/op 2014-06-29 19:26:32,708 DEBUG [pool-1-thread-1-EventThread] procedure.ProcedureMember(155): Submitting new Subprocedure:op 2014-06-29 19:26:32,709 DEBUG [pool-1-thread-1-EventThread] procedure.TestZKProcedure$3(173): Task size:4, getting:2 2014-06-29 19:26:32,709 DEBUG [pool-1-thread-1-EventThread] procedure.ProcedureMember(155): Submitting new Subprocedure:op 2014-06-29 19:26:32,710 DEBUG [member: 'three' subprocedure-pool6-thread-1] procedure.Subprocedure(152): Starting subprocedure 'op' with timeout 10000ms 2014-06-29 19:26:32,709 DEBUG [pool-1-thread-1-EventThread] procedure.ProcedureMember(155): Submitting new Subprocedure:op 2014-06-29 19:26:32,710 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(203): start proc data length is 6 2014-06-29 19:26:32,710 DEBUG [member: 'one' subprocedure-pool4-thread-1] procedure.Subprocedure(152): Starting subprocedure 'op' with timeout 10000ms 2014-06-29 19:26:32,710 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(212): Found data for znode:/hbase/coordination test - 4 cohort members/acquired/op 2014-06-29 19:26:32,710 DEBUG [member: 'one' subprocedure-pool4-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 10000 ms 2014-06-29 19:26:32,711 DEBUG [member: 'one' subprocedure-pool4-thread-1] procedure.Subprocedure(160): Subprocedure 'op' starting 'acquire' stage 2014-06-29 19:26:32,710 DEBUG [member: 'three' subprocedure-pool6-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 10000 ms 2014-06-29 19:26:32,710 DEBUG [member: 'four' subprocedure-pool7-thread-1] procedure.Subprocedure(152): Starting subprocedure 'op' with timeout 10000ms 2014-06-29 19:26:32,711 DEBUG [member: 'three' subprocedure-pool6-thread-1] procedure.Subprocedure(160): Subprocedure 'op' starting 'acquire' stage 2014-06-29 19:26:32,711 DEBUG [pool-1-thread-1-EventThread] procedure.TestZKProcedure$3(173): Task size:4, getting:3 2014-06-29 19:26:32,711 DEBUG [member: 'four' subprocedure-pool7-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 10000 ms 2014-06-29 19:26:32,712 DEBUG [pool-1-thread-1-EventThread] procedure.ProcedureMember(155): Submitting new Subprocedure:op 2014-06-29 19:26:32,712 DEBUG [member: 'one' subprocedure-pool4-thread-1] procedure.Subprocedure(162): Subprocedure 'op' locally acquired 2014-06-29 19:26:32,712 DEBUG [member: 'four' subprocedure-pool7-thread-1] procedure.Subprocedure(160): Subprocedure 'op' starting 'acquire' stage 2014-06-29 19:26:32,712 DEBUG [member: 'three' subprocedure-pool6-thread-1] procedure.Subprocedure(162): Subprocedure 'op' locally acquired 2014-06-29 19:26:32,712 DEBUG [member: 'two' subprocedure-pool5-thread-1] procedure.Subprocedure(152): Starting subprocedure 'op' with timeout 10000ms 2014-06-29 19:26:32,712 DEBUG [member: 'one' subprocedure-pool4-thread-1] procedure.ZKProcedureMemberRpcs(237): Member: 'two' joining acquired barrier for procedure (op) in zk 2014-06-29 19:26:32,713 DEBUG [member: 'two' subprocedure-pool5-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 10000 ms 2014-06-29 19:26:32,712 DEBUG [member: 'three' subprocedure-pool6-thread-1] procedure.ZKProcedureMemberRpcs(237): Member: 'one' joining acquired barrier for procedure (op) in zk 2014-06-29 19:26:32,713 DEBUG [member: 'two' subprocedure-pool5-thread-1] procedure.Subprocedure(160): Subprocedure 'op' starting 'acquire' stage 2014-06-29 19:26:32,714 DEBUG [member: 'four' subprocedure-pool7-thread-1] procedure.Subprocedure(162): Subprocedure 'op' locally acquired 2014-06-29 19:26:32,714 DEBUG [member: 'two' subprocedure-pool5-thread-1] procedure.Subprocedure(162): Subprocedure 'op' locally acquired 2014-06-29 19:26:32,714 DEBUG [member: 'two' subprocedure-pool5-thread-1] procedure.ZKProcedureMemberRpcs(237): Member: 'four' joining acquired barrier for procedure (op) in zk 2014-06-29 19:26:32,715 DEBUG [member: 'four' subprocedure-pool7-thread-1] procedure.ZKProcedureMemberRpcs(237): Member: 'three' joining acquired barrier for procedure (op) in zk 2014-06-29 19:26:32,723 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility-0x146e917cf2b0002, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/coordination test - 4 cohort members/acquired/op/two 2014-06-29 19:26:32,723 DEBUG [member: 'one' subprocedure-pool4-thread-1] procedure.ZKProcedureMemberRpcs(245): Watch for global barrier reached:/hbase/coordination test - 4 cohort members/reached/op 2014-06-29 19:26:32,723 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(170): Node created: /hbase/coordination test - 4 cohort members/acquired/op/two 2014-06-29 19:26:32,723 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2014-06-29 19:26:32,723 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/coordination test - 4 cohort members 2014-06-29 19:26:32,734 DEBUG [member: 'three' subprocedure-pool6-thread-1] procedure.ZKProcedureMemberRpcs(245): Watch for global barrier reached:/hbase/coordination test - 4 cohort members/reached/op 2014-06-29 19:26:32,735 DEBUG [member: 'one' subprocedure-pool4-thread-1] zookeeper.ZKUtil(430): testing utility-0x146e917cf2b0004, quorum=localhost:51439, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/coordination test - 4 cohort members/reached/op 2014-06-29 19:26:32,735 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2014-06-29 19:26:32,735 DEBUG [member: 'one' subprocedure-pool4-thread-1] procedure.Subprocedure(166): Subprocedure 'op' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2014-06-29 19:26:32,745 DEBUG [member: 'two' subprocedure-pool5-thread-1] procedure.ZKProcedureMemberRpcs(245): Watch for global barrier reached:/hbase/coordination test - 4 cohort members/reached/op 2014-06-29 19:26:32,745 DEBUG [member: 'three' subprocedure-pool6-thread-1] zookeeper.ZKUtil(430): testing utility-0x146e917cf2b0003, quorum=localhost:51439, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/coordination test - 4 cohort members/reached/op 2014-06-29 19:26:32,745 DEBUG [member: 'four' subprocedure-pool7-thread-1] procedure.ZKProcedureMemberRpcs(245): Watch for global barrier reached:/hbase/coordination test - 4 cohort members/reached/op 2014-06-29 19:26:32,746 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |----op 2014-06-29 19:26:32,745 DEBUG [member: 'three' subprocedure-pool6-thread-1] procedure.Subprocedure(166): Subprocedure 'op' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2014-06-29 19:26:32,746 DEBUG [member: 'two' subprocedure-pool5-thread-1] zookeeper.ZKUtil(430): testing utility-0x146e917cf2b0006, quorum=localhost:51439, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/coordination test - 4 cohort members/reached/op 2014-06-29 19:26:32,747 DEBUG [member: 'two' subprocedure-pool5-thread-1] procedure.Subprocedure(166): Subprocedure 'op' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2014-06-29 19:26:32,747 DEBUG [member: 'four' subprocedure-pool7-thread-1] zookeeper.ZKUtil(430): testing utility-0x146e917cf2b0005, quorum=localhost:51439, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/coordination test - 4 cohort members/reached/op 2014-06-29 19:26:32,747 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-------three 2014-06-29 19:26:32,747 DEBUG [member: 'four' subprocedure-pool7-thread-1] procedure.Subprocedure(166): Subprocedure 'op' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2014-06-29 19:26:32,748 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-------two 2014-06-29 19:26:32,748 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-------four 2014-06-29 19:26:32,749 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-------one 2014-06-29 19:26:32,749 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-abort 2014-06-29 19:26:32,750 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-reached 2014-06-29 19:26:32,751 DEBUG [pool-1-thread-1-EventThread] procedure.Procedure(294): member: 'two' joining acquired barrier for procedure 'op' on coordinator 2014-06-29 19:26:32,751 DEBUG [pool-1-thread-1-EventThread] procedure.Procedure(303): Waiting on: java.util.concurrent.CountDownLatch@40ece0[Count = 3] remaining members to acquire global barrier 2014-06-29 19:26:32,751 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility-0x146e917cf2b0002, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/coordination test - 4 cohort members/acquired/op/one 2014-06-29 19:26:32,752 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(170): Node created: /hbase/coordination test - 4 cohort members/acquired/op/one 2014-06-29 19:26:32,752 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2014-06-29 19:26:32,752 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/coordination test - 4 cohort members 2014-06-29 19:26:32,752 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2014-06-29 19:26:32,754 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |----op 2014-06-29 19:26:32,754 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-------three 2014-06-29 19:26:32,755 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-------two 2014-06-29 19:26:32,756 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-------four 2014-06-29 19:26:32,756 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-------one 2014-06-29 19:26:32,757 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-abort 2014-06-29 19:26:32,757 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-reached 2014-06-29 19:26:32,758 DEBUG [pool-1-thread-1-EventThread] procedure.Procedure(294): member: 'one' joining acquired barrier for procedure 'op' on coordinator 2014-06-29 19:26:32,758 DEBUG [pool-1-thread-1-EventThread] procedure.Procedure(303): Waiting on: java.util.concurrent.CountDownLatch@40ece0[Count = 2] remaining members to acquire global barrier 2014-06-29 19:26:32,758 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility-0x146e917cf2b0002, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/coordination test - 4 cohort members/acquired/op/four 2014-06-29 19:26:32,759 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(170): Node created: /hbase/coordination test - 4 cohort members/acquired/op/four 2014-06-29 19:26:32,759 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2014-06-29 19:26:32,759 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/coordination test - 4 cohort members 2014-06-29 19:26:32,759 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2014-06-29 19:26:32,760 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |----op 2014-06-29 19:26:32,761 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-------three 2014-06-29 19:26:32,761 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-------two 2014-06-29 19:26:32,762 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-------four 2014-06-29 19:26:32,763 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-------one 2014-06-29 19:26:32,763 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-abort 2014-06-29 19:26:32,764 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-reached 2014-06-29 19:26:32,765 DEBUG [pool-1-thread-1-EventThread] procedure.Procedure(294): member: 'four' joining acquired barrier for procedure 'op' on coordinator 2014-06-29 19:26:32,765 DEBUG [pool-1-thread-1-EventThread] procedure.Procedure(303): Waiting on: java.util.concurrent.CountDownLatch@40ece0[Count = 1] remaining members to acquire global barrier 2014-06-29 19:26:32,765 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility-0x146e917cf2b0002, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/coordination test - 4 cohort members/acquired/op/three 2014-06-29 19:26:32,765 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(170): Node created: /hbase/coordination test - 4 cohort members/acquired/op/three 2014-06-29 19:26:32,765 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2014-06-29 19:26:32,765 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/coordination test - 4 cohort members 2014-06-29 19:26:32,766 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2014-06-29 19:26:32,767 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |----op 2014-06-29 19:26:32,767 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-------three 2014-06-29 19:26:32,768 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-------two 2014-06-29 19:26:32,769 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-------four 2014-06-29 19:26:32,769 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-------one 2014-06-29 19:26:32,770 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-abort 2014-06-29 19:26:32,770 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-reached 2014-06-29 19:26:32,771 DEBUG [pool-1-thread-1-EventThread] procedure.Procedure(294): member: 'three' joining acquired barrier for procedure 'op' on coordinator 2014-06-29 19:26:32,771 DEBUG [pool-1-thread-1-EventThread] procedure.Procedure(303): Waiting on: java.util.concurrent.CountDownLatch@40ece0[Count = 0] remaining members to acquire global barrier 2014-06-29 19:26:32,771 DEBUG [(coordinator)-proc-coordinator-pool3-thread-1] procedure.Procedure(209): Procedure 'op' starting 'in-barrier' execution. 2014-06-29 19:26:32,772 DEBUG [(coordinator)-proc-coordinator-pool3-thread-1] procedure.ZKProcedureCoordinatorRpcs(115): Creating reached barrier zk node:/hbase/coordination test - 4 cohort members/reached/op 2014-06-29 19:26:32,789 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility-0x146e917cf2b0003, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/coordination test - 4 cohort members/reached/op 2014-06-29 19:26:32,789 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility-0x146e917cf2b0006, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/coordination test - 4 cohort members/reached/op 2014-06-29 19:26:32,790 INFO [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs$1(79): Received created event:/hbase/coordination test - 4 cohort members/reached/op 2014-06-29 19:26:32,789 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility-0x146e917cf2b0005, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/coordination test - 4 cohort members/reached/op 2014-06-29 19:26:32,789 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility-0x146e917cf2b0004, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/coordination test - 4 cohort members/reached/op 2014-06-29 19:26:32,790 DEBUG [(coordinator)-proc-coordinator-pool3-thread-1] zookeeper.ZKUtil(430): testing utility-0x146e917cf2b0002, quorum=localhost:51439, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/coordination test - 4 cohort members/reached/op/two 2014-06-29 19:26:32,790 INFO [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs$1(79): Received created event:/hbase/coordination test - 4 cohort members/reached/op 2014-06-29 19:26:32,790 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(130): Recieved reached global barrier:/hbase/coordination test - 4 cohort members/reached/op 2014-06-29 19:26:32,790 INFO [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs$1(79): Received created event:/hbase/coordination test - 4 cohort members/reached/op 2014-06-29 19:26:32,791 DEBUG [member: 'one' subprocedure-pool4-thread-1] procedure.Subprocedure(180): Subprocedure 'op' received 'reached' from coordinator. 2014-06-29 19:26:32,791 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(130): Recieved reached global barrier:/hbase/coordination test - 4 cohort members/reached/op 2014-06-29 19:26:32,790 INFO [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs$1(79): Received created event:/hbase/coordination test - 4 cohort members/reached/op 2014-06-29 19:26:32,791 DEBUG [member: 'three' subprocedure-pool6-thread-1] procedure.Subprocedure(180): Subprocedure 'op' received 'reached' from coordinator. 2014-06-29 19:26:32,791 DEBUG [member: 'one' subprocedure-pool4-thread-1] procedure.Subprocedure(182): Subprocedure 'op' locally completed 2014-06-29 19:26:32,791 DEBUG [(coordinator)-proc-coordinator-pool3-thread-1] zookeeper.ZKUtil(430): testing utility-0x146e917cf2b0002, quorum=localhost:51439, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/coordination test - 4 cohort members/reached/op/one 2014-06-29 19:26:32,791 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(130): Recieved reached global barrier:/hbase/coordination test - 4 cohort members/reached/op 2014-06-29 19:26:32,792 DEBUG [member: 'one' subprocedure-pool4-thread-1] procedure.ZKProcedureMemberRpcs(261): Marking procedure 'op' completed for member 'two' in zk 2014-06-29 19:26:32,792 DEBUG [member: 'three' subprocedure-pool6-thread-1] procedure.Subprocedure(182): Subprocedure 'op' locally completed 2014-06-29 19:26:32,791 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(130): Recieved reached global barrier:/hbase/coordination test - 4 cohort members/reached/op 2014-06-29 19:26:32,792 DEBUG [(coordinator)-proc-coordinator-pool3-thread-1] zookeeper.ZKUtil(430): testing utility-0x146e917cf2b0002, quorum=localhost:51439, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/coordination test - 4 cohort members/reached/op/four 2014-06-29 19:26:32,792 DEBUG [member: 'three' subprocedure-pool6-thread-1] procedure.ZKProcedureMemberRpcs(261): Marking procedure 'op' completed for member 'one' in zk 2014-06-29 19:26:32,792 DEBUG [member: 'four' subprocedure-pool7-thread-1] procedure.Subprocedure(180): Subprocedure 'op' received 'reached' from coordinator. 2014-06-29 19:26:32,793 DEBUG [member: 'two' subprocedure-pool5-thread-1] procedure.Subprocedure(180): Subprocedure 'op' received 'reached' from coordinator. 2014-06-29 19:26:32,793 DEBUG [member: 'four' subprocedure-pool7-thread-1] procedure.Subprocedure(182): Subprocedure 'op' locally completed 2014-06-29 19:26:32,793 DEBUG [member: 'two' subprocedure-pool5-thread-1] procedure.Subprocedure(182): Subprocedure 'op' locally completed 2014-06-29 19:26:32,793 DEBUG [member: 'four' subprocedure-pool7-thread-1] procedure.ZKProcedureMemberRpcs(261): Marking procedure 'op' completed for member 'three' in zk 2014-06-29 19:26:32,794 DEBUG [member: 'two' subprocedure-pool5-thread-1] procedure.ZKProcedureMemberRpcs(261): Marking procedure 'op' completed for member 'four' in zk 2014-06-29 19:26:32,794 DEBUG [(coordinator)-proc-coordinator-pool3-thread-1] zookeeper.ZKUtil(430): testing utility-0x146e917cf2b0002, quorum=localhost:51439, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/coordination test - 4 cohort members/reached/op/three 2014-06-29 19:26:33,487 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility-0x146e917cf2b0002, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/coordination test - 4 cohort members/reached/op/two 2014-06-29 19:26:33,488 DEBUG [member: 'one' subprocedure-pool4-thread-1] procedure.Subprocedure(186): Subprocedure 'op' has notified controller of completion 2014-06-29 19:26:33,488 DEBUG [member: 'three' subprocedure-pool6-thread-1] procedure.Subprocedure(186): Subprocedure 'op' has notified controller of completion 2014-06-29 19:26:33,488 DEBUG [member: 'one' subprocedure-pool4-thread-1] errorhandling.TimeoutExceptionInjector(89): Marking timer as complete - no error notifications will be received for this timer. 2014-06-29 19:26:33,488 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(170): Node created: /hbase/coordination test - 4 cohort members/reached/op/two 2014-06-29 19:26:33,488 DEBUG [member: 'one' subprocedure-pool4-thread-1] procedure.Subprocedure(211): Subprocedure 'op' completed. 2014-06-29 19:26:33,488 DEBUG [member: 'three' subprocedure-pool6-thread-1] errorhandling.TimeoutExceptionInjector(89): Marking timer as complete - no error notifications will be received for this timer. 2014-06-29 19:26:33,489 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2014-06-29 19:26:33,489 DEBUG [member: 'three' subprocedure-pool6-thread-1] procedure.Subprocedure(211): Subprocedure 'op' completed. 2014-06-29 19:26:33,489 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/coordination test - 4 cohort members 2014-06-29 19:26:33,544 DEBUG [member: 'four' subprocedure-pool7-thread-1] procedure.Subprocedure(186): Subprocedure 'op' has notified controller of completion 2014-06-29 19:26:33,544 DEBUG [member: 'four' subprocedure-pool7-thread-1] errorhandling.TimeoutExceptionInjector(89): Marking timer as complete - no error notifications will be received for this timer. 2014-06-29 19:26:33,544 DEBUG [member: 'four' subprocedure-pool7-thread-1] procedure.Subprocedure(211): Subprocedure 'op' completed. 2014-06-29 19:26:33,580 DEBUG [member: 'two' subprocedure-pool5-thread-1] procedure.Subprocedure(186): Subprocedure 'op' has notified controller of completion 2014-06-29 19:26:33,580 DEBUG [member: 'two' subprocedure-pool5-thread-1] errorhandling.TimeoutExceptionInjector(89): Marking timer as complete - no error notifications will be received for this timer. 2014-06-29 19:26:33,580 DEBUG [member: 'two' subprocedure-pool5-thread-1] procedure.Subprocedure(211): Subprocedure 'op' completed. 2014-06-29 19:26:33,581 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2014-06-29 19:26:33,581 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |----op 2014-06-29 19:26:33,582 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-------three 2014-06-29 19:26:33,583 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-------two 2014-06-29 19:26:33,583 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-------four 2014-06-29 19:26:33,584 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-------one 2014-06-29 19:26:33,584 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-abort 2014-06-29 19:26:33,585 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-reached 2014-06-29 19:26:33,586 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |----op 2014-06-29 19:26:33,586 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-------three 2014-06-29 19:26:33,587 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-------two 2014-06-29 19:26:33,587 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-------four 2014-06-29 19:26:33,588 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-------one 2014-06-29 19:26:33,589 DEBUG [pool-1-thread-1-EventThread] procedure.Procedure(324): Member: 'two' released barrier for procedure'op', counting down latch. Waiting for 3 more 2014-06-29 19:26:33,589 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility-0x146e917cf2b0002, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/coordination test - 4 cohort members/reached/op/one 2014-06-29 19:26:33,589 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(170): Node created: /hbase/coordination test - 4 cohort members/reached/op/one 2014-06-29 19:26:33,589 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2014-06-29 19:26:33,589 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/coordination test - 4 cohort members 2014-06-29 19:26:33,590 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2014-06-29 19:26:33,591 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |----op 2014-06-29 19:26:33,591 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-------three 2014-06-29 19:26:33,592 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-------two 2014-06-29 19:26:33,593 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-------four 2014-06-29 19:26:33,593 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-------one 2014-06-29 19:26:33,594 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-abort 2014-06-29 19:26:33,595 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-reached 2014-06-29 19:26:33,595 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |----op 2014-06-29 19:26:33,596 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-------three 2014-06-29 19:26:33,596 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-------two 2014-06-29 19:26:33,597 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-------four 2014-06-29 19:26:33,598 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-------one 2014-06-29 19:26:33,598 DEBUG [pool-1-thread-1-EventThread] procedure.Procedure(324): Member: 'one' released barrier for procedure'op', counting down latch. Waiting for 2 more 2014-06-29 19:26:33,598 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility-0x146e917cf2b0002, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/coordination test - 4 cohort members/reached/op/three 2014-06-29 19:26:33,599 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(170): Node created: /hbase/coordination test - 4 cohort members/reached/op/three 2014-06-29 19:26:33,599 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2014-06-29 19:26:33,599 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/coordination test - 4 cohort members 2014-06-29 19:26:33,599 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2014-06-29 19:26:33,600 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |----op 2014-06-29 19:26:33,601 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-------three 2014-06-29 19:26:33,601 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-------two 2014-06-29 19:26:33,602 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-------four 2014-06-29 19:26:33,603 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-------one 2014-06-29 19:26:33,603 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-abort 2014-06-29 19:26:33,604 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-reached 2014-06-29 19:26:33,605 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |----op 2014-06-29 19:26:33,606 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-------three 2014-06-29 19:26:33,606 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-------two 2014-06-29 19:26:33,607 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-------four 2014-06-29 19:26:33,607 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-------one 2014-06-29 19:26:33,608 DEBUG [pool-1-thread-1-EventThread] procedure.Procedure(324): Member: 'three' released barrier for procedure'op', counting down latch. Waiting for 1 more 2014-06-29 19:26:33,608 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility-0x146e917cf2b0002, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/coordination test - 4 cohort members/reached/op/four 2014-06-29 19:26:33,608 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(170): Node created: /hbase/coordination test - 4 cohort members/reached/op/four 2014-06-29 19:26:33,609 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2014-06-29 19:26:33,609 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/coordination test - 4 cohort members 2014-06-29 19:26:33,609 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2014-06-29 19:26:33,610 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |----op 2014-06-29 19:26:33,611 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-------three 2014-06-29 19:26:33,611 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-------two 2014-06-29 19:26:33,612 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-------four 2014-06-29 19:26:33,613 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-------one 2014-06-29 19:26:33,613 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-abort 2014-06-29 19:26:33,614 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-reached 2014-06-29 19:26:33,615 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |----op 2014-06-29 19:26:33,615 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-------three 2014-06-29 19:26:33,616 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-------two 2014-06-29 19:26:33,617 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-------four 2014-06-29 19:26:33,617 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-------one 2014-06-29 19:26:33,618 DEBUG [pool-1-thread-1-EventThread] procedure.Procedure(324): Member: 'four' released barrier for procedure'op', counting down latch. Waiting for 0 more 2014-06-29 19:26:33,618 INFO [(coordinator)-proc-coordinator-pool3-thread-1] procedure.Procedure(217): Procedure 'op' execution completed 2014-06-29 19:26:33,618 DEBUG [(coordinator)-proc-coordinator-pool3-thread-1] procedure.Procedure(226): Running finish phase. 2014-06-29 19:26:33,618 DEBUG [(coordinator)-proc-coordinator-pool3-thread-1] procedure.Procedure(277): Finished coordinator procedure - removing self from list of running procedures 2014-06-29 19:26:33,619 DEBUG [(coordinator)-proc-coordinator-pool3-thread-1] procedure.ZKProcedureCoordinatorRpcs(141): Attempting to clean out zk node for op:op 2014-06-29 19:26:33,619 INFO [(coordinator)-proc-coordinator-pool3-thread-1] procedure.ZKProcedureUtil(284): Clearing all znodes for procedure opincluding nodes /hbase/coordination test - 4 cohort members/acquired /hbase/coordination test - 4 cohort members/reached /hbase/coordination test - 4 cohort members/abort 2014-06-29 19:26:33,689 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility-0x146e917cf2b0005, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/coordination test - 4 cohort members/acquired 2014-06-29 19:26:33,689 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility-0x146e917cf2b0004, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/coordination test - 4 cohort members/acquired 2014-06-29 19:26:33,689 INFO [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs$1(106): Received procedure start children changed event: /hbase/coordination test - 4 cohort members/acquired 2014-06-29 19:26:33,689 INFO [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs$1(106): Received procedure start children changed event: /hbase/coordination test - 4 cohort members/acquired 2014-06-29 19:26:33,689 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/coordination test - 4 cohort members/acquired' 2014-06-29 19:26:33,689 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility-0x146e917cf2b0003, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/coordination test - 4 cohort members/acquired 2014-06-29 19:26:33,689 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility-0x146e917cf2b0006, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/coordination test - 4 cohort members/acquired 2014-06-29 19:26:33,690 INFO [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs$1(106): Received procedure start children changed event: /hbase/coordination test - 4 cohort members/acquired 2014-06-29 19:26:33,690 INFO [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs$1(106): Received procedure start children changed event: /hbase/coordination test - 4 cohort members/acquired 2014-06-29 19:26:33,690 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/coordination test - 4 cohort members/acquired' 2014-06-29 19:26:33,690 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/coordination test - 4 cohort members/acquired' 2014-06-29 19:26:33,690 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/coordination test - 4 cohort members/acquired' 2014-06-29 19:26:33,748 DEBUG [(coordinator)-proc-coordinator-pool3-thread-1] errorhandling.TimeoutExceptionInjector(89): Marking timer as complete - no error notifications will be received for this timer. 2014-06-29 19:26:33,749 DEBUG [pool-1-thread-1] procedure.TestZKProcedure(392): Checking mock:0 2014-06-29 19:26:33,750 DEBUG [pool-1-thread-1] procedure.TestZKProcedure(392): Checking mock:1 2014-06-29 19:26:33,750 DEBUG [pool-1-thread-1] procedure.TestZKProcedure(392): Checking mock:2 2014-06-29 19:26:33,751 DEBUG [pool-1-thread-1] procedure.TestZKProcedure(392): Checking mock:3 2014-06-29 19:26:33,766 INFO [pool-1-thread-1] hbase.ResourceChecker(171): after: procedure.TestZKProcedure#testMultipleMembers Thread=23 (was 13) - Thread LEAK? -, OpenFileDescriptor=171 (was 144) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=40000 (was 40000), SystemLoadAverage=360 (was 374), ProcessCount=168 (was 168), AvailableMemoryMB=3168 (was 3169), ConnectionCount=0 (was 0) 2014-06-29 19:26:33,781 INFO [pool-1-thread-1] hbase.ResourceChecker(147): before: procedure.TestZKProcedure#testMultiCohortWithMemberTimeoutDuringPrepare Thread=24, OpenFileDescriptor=173, MaxFileDescriptor=40000, SystemLoadAverage=360, ProcessCount=168, AvailableMemoryMB=3168, ConnectionCount=0 2014-06-29 19:26:33,785 INFO [pool-1-thread-1] zookeeper.RecoverableZooKeeper(120): Process identifier=testing utility connecting to ZooKeeper ensemble=localhost:51439 2014-06-29 19:26:33,799 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2014-06-29 19:26:33,801 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(387): testing utility-0x146e917cf2b0007 connected 2014-06-29 19:26:33,882 INFO [pool-1-thread-1] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/error injection coordination/acquired /hbase/error injection coordination/reached /hbase/error injection coordination/abort 2014-06-29 19:26:33,884 DEBUG [pool-1-thread-1] procedure.ZKProcedureCoordinatorRpcs(195): Starting the controller for procedure member:coordinator 2014-06-29 19:26:33,897 INFO [pool-1-thread-1] zookeeper.RecoverableZooKeeper(120): Process identifier=testing utility connecting to ZooKeeper ensemble=localhost:51439 2014-06-29 19:26:33,914 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2014-06-29 19:26:33,916 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(387): testing utility-0x146e917cf2b0008 connected 2014-06-29 19:26:33,936 INFO [pool-1-thread-1] zookeeper.RecoverableZooKeeper(508): Node /hbase/error injection coordination/acquired already exists and this is not a retry 2014-06-29 19:26:33,938 DEBUG [pool-1-thread-1] procedure.ZKProcedureMemberRpcs(337): Starting procedure member 'one' 2014-06-29 19:26:33,938 DEBUG [pool-1-thread-1] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/error injection coordination/abort' 2014-06-29 19:26:33,939 DEBUG [pool-1-thread-1] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/error injection coordination/acquired' 2014-06-29 19:26:33,942 INFO [pool-1-thread-1] zookeeper.RecoverableZooKeeper(120): Process identifier=testing utility connecting to ZooKeeper ensemble=localhost:51439 2014-06-29 19:26:33,959 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2014-06-29 19:26:33,960 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(387): testing utility-0x146e917cf2b0009 connected 2014-06-29 19:26:33,980 INFO [pool-1-thread-1] zookeeper.RecoverableZooKeeper(508): Node /hbase/error injection coordination/acquired already exists and this is not a retry 2014-06-29 19:26:33,982 DEBUG [pool-1-thread-1] procedure.ZKProcedureMemberRpcs(337): Starting procedure member 'two' 2014-06-29 19:26:33,982 DEBUG [pool-1-thread-1] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/error injection coordination/abort' 2014-06-29 19:26:33,983 DEBUG [pool-1-thread-1] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/error injection coordination/acquired' 2014-06-29 19:26:33,986 INFO [pool-1-thread-1] zookeeper.RecoverableZooKeeper(120): Process identifier=testing utility connecting to ZooKeeper ensemble=localhost:51439 2014-06-29 19:26:34,004 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2014-06-29 19:26:34,005 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(387): testing utility-0x146e917cf2b000a connected 2014-06-29 19:26:34,034 INFO [pool-1-thread-1] zookeeper.RecoverableZooKeeper(508): Node /hbase/error injection coordination/acquired already exists and this is not a retry 2014-06-29 19:26:34,036 DEBUG [pool-1-thread-1] procedure.ZKProcedureMemberRpcs(337): Starting procedure member 'three' 2014-06-29 19:26:34,036 DEBUG [pool-1-thread-1] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/error injection coordination/abort' 2014-06-29 19:26:34,037 DEBUG [pool-1-thread-1] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/error injection coordination/acquired' 2014-06-29 19:26:34,065 INFO [(coordinator)-proc-coordinator-pool8-thread-1] procedure.Procedure(193): Starting procedure 'op' 2014-06-29 19:26:34,065 DEBUG [(coordinator)-proc-coordinator-pool8-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 10000 ms 2014-06-29 19:26:34,077 DEBUG [(coordinator)-proc-coordinator-pool8-thread-1] procedure.Procedure(201): Procedure 'op' starting 'acquire' 2014-06-29 19:26:34,077 DEBUG [(coordinator)-proc-coordinator-pool8-thread-1] procedure.Procedure(243): Starting procedure 'op', kicking off acquire phase on members. 2014-06-29 19:26:34,078 DEBUG [(coordinator)-proc-coordinator-pool8-thread-1] zookeeper.ZKUtil(430): testing utility-0x146e917cf2b0007, quorum=localhost:51439, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/error injection coordination/abort/op 2014-06-29 19:26:34,078 DEBUG [(coordinator)-proc-coordinator-pool8-thread-1] procedure.ZKProcedureCoordinatorRpcs(93): Creating acquire znode:/hbase/error injection coordination/acquired/op 2014-06-29 19:26:34,092 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility-0x146e917cf2b000a, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/error injection coordination/acquired 2014-06-29 19:26:34,092 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility-0x146e917cf2b0009, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/error injection coordination/acquired 2014-06-29 19:26:34,092 DEBUG [(coordinator)-proc-coordinator-pool8-thread-1] procedure.ZKProcedureCoordinatorRpcs(101): Watching for acquire node:/hbase/error injection coordination/acquired/op/one 2014-06-29 19:26:34,092 INFO [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs$1(106): Received procedure start children changed event: /hbase/error injection coordination/acquired 2014-06-29 19:26:34,092 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/error injection coordination/acquired' 2014-06-29 19:26:34,092 INFO [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs$1(106): Received procedure start children changed event: /hbase/error injection coordination/acquired 2014-06-29 19:26:34,092 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility-0x146e917cf2b0008, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/error injection coordination/acquired 2014-06-29 19:26:34,093 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/error injection coordination/acquired' 2014-06-29 19:26:34,093 INFO [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs$1(106): Received procedure start children changed event: /hbase/error injection coordination/acquired 2014-06-29 19:26:34,093 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/error injection coordination/acquired' 2014-06-29 19:26:34,094 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(184): Found procedure znode: /hbase/error injection coordination/acquired/op 2014-06-29 19:26:34,094 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(184): Found procedure znode: /hbase/error injection coordination/acquired/op 2014-06-29 19:26:34,094 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(184): Found procedure znode: /hbase/error injection coordination/acquired/op 2014-06-29 19:26:34,094 DEBUG [(coordinator)-proc-coordinator-pool8-thread-1] zookeeper.ZKUtil(430): testing utility-0x146e917cf2b0007, quorum=localhost:51439, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/error injection coordination/acquired/op/one 2014-06-29 19:26:34,094 DEBUG [(coordinator)-proc-coordinator-pool8-thread-1] procedure.ZKProcedureCoordinatorRpcs(101): Watching for acquire node:/hbase/error injection coordination/acquired/op/two 2014-06-29 19:26:34,095 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(430): testing utility-0x146e917cf2b000a, quorum=localhost:51439, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/error injection coordination/abort/op 2014-06-29 19:26:34,095 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(430): testing utility-0x146e917cf2b0009, quorum=localhost:51439, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/error injection coordination/abort/op 2014-06-29 19:26:34,095 DEBUG [(coordinator)-proc-coordinator-pool8-thread-1] zookeeper.ZKUtil(430): testing utility-0x146e917cf2b0007, quorum=localhost:51439, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/error injection coordination/acquired/op/two 2014-06-29 19:26:34,095 DEBUG [(coordinator)-proc-coordinator-pool8-thread-1] procedure.ZKProcedureCoordinatorRpcs(101): Watching for acquire node:/hbase/error injection coordination/acquired/op/three 2014-06-29 19:26:34,095 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(430): testing utility-0x146e917cf2b0008, quorum=localhost:51439, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/error injection coordination/abort/op 2014-06-29 19:26:34,096 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(203): start proc data length is 6 2014-06-29 19:26:34,096 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(203): start proc data length is 6 2014-06-29 19:26:34,096 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(212): Found data for znode:/hbase/error injection coordination/acquired/op 2014-06-29 19:26:34,096 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(212): Found data for znode:/hbase/error injection coordination/acquired/op 2014-06-29 19:26:34,096 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(203): start proc data length is 6 2014-06-29 19:26:34,096 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(212): Found data for znode:/hbase/error injection coordination/acquired/op 2014-06-29 19:26:34,097 DEBUG [(coordinator)-proc-coordinator-pool8-thread-1] zookeeper.ZKUtil(430): testing utility-0x146e917cf2b0007, quorum=localhost:51439, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/error injection coordination/acquired/op/three 2014-06-29 19:26:34,097 DEBUG [(coordinator)-proc-coordinator-pool8-thread-1] procedure.Procedure(205): Waiting for all members to 'acquire' 2014-06-29 19:26:34,098 DEBUG [pool-1-thread-1-EventThread] procedure.ProcedureMember(155): Submitting new Subprocedure:op 2014-06-29 19:26:34,098 DEBUG [pool-1-thread-1-EventThread] procedure.ProcedureMember(155): Submitting new Subprocedure:op 2014-06-29 19:26:34,098 DEBUG [member: 'three' subprocedure-pool11-thread-1] procedure.Subprocedure(152): Starting subprocedure 'op' with timeout 10000ms 2014-06-29 19:26:34,098 DEBUG [pool-1-thread-1-EventThread] procedure.ProcedureMember(155): Submitting new Subprocedure:op 2014-06-29 19:26:34,099 DEBUG [member: 'three' subprocedure-pool11-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 10000 ms 2014-06-29 19:26:34,099 DEBUG [member: 'one' subprocedure-pool9-thread-1] procedure.Subprocedure(152): Starting subprocedure 'op' with timeout 10000ms 2014-06-29 19:26:34,099 DEBUG [member: 'three' subprocedure-pool11-thread-1] procedure.Subprocedure(160): Subprocedure 'op' starting 'acquire' stage 2014-06-29 19:26:34,099 WARN [member: 'one' subprocedure-pool9-thread-1] errorhandling.TimeoutExceptionInjector(105): Timer already started, can't be started again. Ignoring second request. 2014-06-29 19:26:34,100 DEBUG [member: 'one' subprocedure-pool9-thread-1] procedure.Subprocedure(160): Subprocedure 'op' starting 'acquire' stage 2014-06-29 19:26:34,099 DEBUG [member: 'two' subprocedure-pool10-thread-1] procedure.Subprocedure(152): Starting subprocedure 'op' with timeout 10000ms 2014-06-29 19:26:34,100 DEBUG [member: 'two' subprocedure-pool10-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 10000 ms 2014-06-29 19:26:34,100 DEBUG [member: 'two' subprocedure-pool10-thread-1] procedure.Subprocedure(160): Subprocedure 'op' starting 'acquire' stage 2014-06-29 19:26:34,100 DEBUG [member: 'three' subprocedure-pool11-thread-1] procedure.Subprocedure(162): Subprocedure 'op' locally acquired 2014-06-29 19:26:34,101 DEBUG [member: 'three' subprocedure-pool11-thread-1] procedure.ZKProcedureMemberRpcs(237): Member: 'one' joining acquired barrier for procedure (op) in zk 2014-06-29 19:26:34,101 DEBUG [member: 'one' subprocedure-pool9-thread-1] procedure.Subprocedure(162): Subprocedure 'op' locally acquired 2014-06-29 19:26:34,101 DEBUG [member: 'one' subprocedure-pool9-thread-1] procedure.ZKProcedureMemberRpcs(237): Member: 'one' joining acquired barrier for procedure (op) in zk 2014-06-29 19:26:34,102 DEBUG [member: 'two' subprocedure-pool10-thread-1] procedure.TestZKProcedure$4(243): Sending error to coordinator 2014-06-29 19:26:34,103 ERROR [member: 'two' subprocedure-pool10-thread-1] procedure.TestZKProcedure$4(247): Remote commit failure, not propagating error:org.apache.hadoop.hbase.errorhandling.TimeoutException via TIMER:org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms 2014-06-29 19:26:34,103 DEBUG [member: 'two' subprocedure-pool10-thread-1] procedure.ProcedureMember(241): Request received to abort procedure op org.apache.hadoop.hbase.errorhandling.TimeoutException via TIMER:org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:244) at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:238) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34) at org.mockito.internal.MockHandler.handle(MockHandler.java:99) at org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36) at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.acquireBarrier() at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:161) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl.call(Subprocedure.java:315) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) Caused by: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms ... 14 more 2014-06-29 19:26:34,105 ERROR [member: 'two' subprocedure-pool10-thread-1] procedure.ProcedureMember(250): Propagating foreign exception to subprocedure op org.apache.hadoop.hbase.errorhandling.TimeoutException via TIMER:org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:244) at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:238) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34) at org.mockito.internal.MockHandler.handle(MockHandler.java:99) at org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36) at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.acquireBarrier() at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:161) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl.call(Subprocedure.java:315) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) Caused by: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms ... 14 more 2014-06-29 19:26:34,106 ERROR [member: 'two' subprocedure-pool10-thread-1] procedure.Subprocedure(265): Propagating foreign exception to subprocedure op org.apache.hadoop.hbase.errorhandling.TimeoutException via TIMER:org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:244) at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:238) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34) at org.mockito.internal.MockHandler.handle(MockHandler.java:99) at org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36) at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.acquireBarrier() at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:161) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl.call(Subprocedure.java:315) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) Caused by: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms ... 14 more 2014-06-29 19:26:34,107 DEBUG [member: 'two' subprocedure-pool10-thread-1] errorhandling.ForeignExceptionDispatcher(66): accepting received exception org.apache.hadoop.hbase.errorhandling.TimeoutException via TIMER:org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:244) at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:238) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34) at org.mockito.internal.MockHandler.handle(MockHandler.java:99) at org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36) at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.acquireBarrier() at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:161) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl.call(Subprocedure.java:315) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) Caused by: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms ... 14 more 2014-06-29 19:26:34,107 DEBUG [member: 'two' subprocedure-pool10-thread-1] procedure.ZKProcedureMemberRpcs(283): Aborting procedure (op) in zk 2014-06-29 19:26:34,120 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility-0x146e917cf2b0007, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/error injection coordination/acquired/op/one 2014-06-29 19:26:34,121 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(170): Node created: /hbase/error injection coordination/acquired/op/one 2014-06-29 19:26:34,121 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2014-06-29 19:26:34,122 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/error injection coordination 2014-06-29 19:26:34,122 DEBUG [member: 'three' subprocedure-pool11-thread-1] procedure.ZKProcedureMemberRpcs(245): Watch for global barrier reached:/hbase/error injection coordination/reached/op 2014-06-29 19:26:34,137 INFO [member: 'one' subprocedure-pool9-thread-1] zookeeper.RecoverableZooKeeper(508): Node /hbase/error injection coordination/acquired/op/one already exists and this is not a retry 2014-06-29 19:26:34,137 DEBUG [member: 'three' subprocedure-pool11-thread-1] zookeeper.ZKUtil(430): testing utility-0x146e917cf2b0008, quorum=localhost:51439, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/error injection coordination/reached/op 2014-06-29 19:26:34,137 DEBUG [member: 'three' subprocedure-pool11-thread-1] procedure.Subprocedure(166): Subprocedure 'op' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2014-06-29 19:26:34,137 DEBUG [member: 'one' subprocedure-pool9-thread-1] procedure.ZKProcedureMemberRpcs(245): Watch for global barrier reached:/hbase/error injection coordination/reached/op 2014-06-29 19:26:34,137 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2014-06-29 19:26:34,138 DEBUG [member: 'one' subprocedure-pool9-thread-1] zookeeper.ZKUtil(430): testing utility-0x146e917cf2b0008, quorum=localhost:51439, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/error injection coordination/reached/op 2014-06-29 19:26:34,138 DEBUG [member: 'one' subprocedure-pool9-thread-1] procedure.Subprocedure(166): Subprocedure 'op' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2014-06-29 19:26:34,343 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility-0x146e917cf2b000a, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/error injection coordination/abort/op 2014-06-29 19:26:34,344 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility-0x146e917cf2b0009, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/error injection coordination/abort/op 2014-06-29 19:26:34,344 DEBUG [member: 'two' subprocedure-pool10-thread-1] procedure.ZKProcedureMemberRpcs(289): Finished creating abort znode:/hbase/error injection coordination/abort/op 2014-06-29 19:26:34,344 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |----op 2014-06-29 19:26:34,344 INFO [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs$1(79): Received created event:/hbase/error injection coordination/abort/op 2014-06-29 19:26:34,344 INFO [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs$1(79): Received created event:/hbase/error injection coordination/abort/op 2014-06-29 19:26:34,344 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(304): Aborting procedure member for znode /hbase/error injection coordination/abort/op 2014-06-29 19:26:34,345 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(304): Aborting procedure member for znode /hbase/error injection coordination/abort/op 2014-06-29 19:26:34,344 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility-0x146e917cf2b0008, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/error injection coordination/abort/op 2014-06-29 19:26:34,345 INFO [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs$1(79): Received created event:/hbase/error injection coordination/abort/op 2014-06-29 19:26:34,345 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-------one 2014-06-29 19:26:34,345 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(304): Aborting procedure member for znode /hbase/error injection coordination/abort/op 2014-06-29 19:26:34,352 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-abort 2014-06-29 19:26:34,352 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |----op 2014-06-29 19:26:34,352 DEBUG [pool-1-thread-1-EventThread] procedure.ProcedureMember(241): Request received to abort procedure op org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via TIMER:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.errorhandling.ForeignException.deserialize(ForeignException.java:171) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.abort(ZKProcedureMemberRpcs.java:320) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs$1.nodeCreated(ZKProcedureMemberRpcs.java:94) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:327) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) Caused by: org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:244) at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:238) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34) at org.mockito.internal.MockHandler.handle(MockHandler.java:99) at org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36) at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.acquireBarrier() at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:161) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl.call(Subprocedure.java:315) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) 2014-06-29 19:26:34,353 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-reached 2014-06-29 19:26:34,353 ERROR [pool-1-thread-1-EventThread] procedure.ProcedureMember(250): Propagating foreign exception to subprocedure op org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via TIMER:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.errorhandling.ForeignException.deserialize(ForeignException.java:171) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.abort(ZKProcedureMemberRpcs.java:320) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs$1.nodeCreated(ZKProcedureMemberRpcs.java:94) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:327) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) Caused by: org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:244) at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:238) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34) at org.mockito.internal.MockHandler.handle(MockHandler.java:99) at org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36) at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.acquireBarrier() at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:161) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl.call(Subprocedure.java:315) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) 2014-06-29 19:26:34,353 DEBUG [pool-1-thread-1-EventThread] procedure.ProcedureMember(241): Request received to abort procedure op org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via TIMER:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.errorhandling.ForeignException.deserialize(ForeignException.java:171) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.abort(ZKProcedureMemberRpcs.java:320) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs$1.nodeCreated(ZKProcedureMemberRpcs.java:94) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:327) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) Caused by: org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:244) at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:238) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34) at org.mockito.internal.MockHandler.handle(MockHandler.java:99) at org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36) at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.acquireBarrier() at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:161) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl.call(Subprocedure.java:315) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) 2014-06-29 19:26:34,354 ERROR [pool-1-thread-1-EventThread] procedure.Subprocedure(265): Propagating foreign exception to subprocedure op org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via TIMER:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.errorhandling.ForeignException.deserialize(ForeignException.java:171) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.abort(ZKProcedureMemberRpcs.java:320) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs$1.nodeCreated(ZKProcedureMemberRpcs.java:94) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:327) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) Caused by: org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:244) at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:238) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34) at org.mockito.internal.MockHandler.handle(MockHandler.java:99) at org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36) at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.acquireBarrier() at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:161) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl.call(Subprocedure.java:315) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) 2014-06-29 19:26:34,355 DEBUG [pool-1-thread-1-EventThread] procedure.ProcedureMember(241): Request received to abort procedure op org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via TIMER:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.errorhandling.ForeignException.deserialize(ForeignException.java:171) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.abort(ZKProcedureMemberRpcs.java:320) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs$1.nodeCreated(ZKProcedureMemberRpcs.java:94) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:327) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) Caused by: org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:244) at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:238) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34) at org.mockito.internal.MockHandler.handle(MockHandler.java:99) at org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36) at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.acquireBarrier() at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:161) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl.call(Subprocedure.java:315) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) 2014-06-29 19:26:34,355 DEBUG [pool-1-thread-1-EventThread] errorhandling.ForeignExceptionDispatcher(66): accepting received exception org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via TIMER:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.errorhandling.ForeignException.deserialize(ForeignException.java:171) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.abort(ZKProcedureMemberRpcs.java:320) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs$1.nodeCreated(ZKProcedureMemberRpcs.java:94) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:327) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) Caused by: org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:244) at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:238) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34) at org.mockito.internal.MockHandler.handle(MockHandler.java:99) at org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36) at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.acquireBarrier() at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:161) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl.call(Subprocedure.java:315) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) 2014-06-29 19:26:34,354 ERROR [pool-1-thread-1-EventThread] procedure.ProcedureMember(250): Propagating foreign exception to subprocedure op org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via TIMER:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.errorhandling.ForeignException.deserialize(ForeignException.java:171) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.abort(ZKProcedureMemberRpcs.java:320) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs$1.nodeCreated(ZKProcedureMemberRpcs.java:94) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:327) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) Caused by: org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:244) at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:238) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34) at org.mockito.internal.MockHandler.handle(MockHandler.java:99) at org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36) at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.acquireBarrier() at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:161) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl.call(Subprocedure.java:315) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) 2014-06-29 19:26:34,354 DEBUG [pool-1-thread-1-EventThread] procedure.Procedure(294): member: 'one' joining acquired barrier for procedure 'op' on coordinator 2014-06-29 19:26:34,356 ERROR [pool-1-thread-1-EventThread] procedure.Subprocedure(265): Propagating foreign exception to subprocedure op org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via TIMER:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.errorhandling.ForeignException.deserialize(ForeignException.java:171) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.abort(ZKProcedureMemberRpcs.java:320) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs$1.nodeCreated(ZKProcedureMemberRpcs.java:94) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:327) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) Caused by: org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:244) at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:238) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34) at org.mockito.internal.MockHandler.handle(MockHandler.java:99) at org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36) at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.acquireBarrier() at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:161) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl.call(Subprocedure.java:315) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) 2014-06-29 19:26:34,356 DEBUG [pool-1-thread-1-EventThread] procedure.Subprocedure$1(106): Was remote foreign exception, not redispatching error org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via TIMER:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.errorhandling.ForeignException.deserialize(ForeignException.java:171) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.abort(ZKProcedureMemberRpcs.java:320) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs$1.nodeCreated(ZKProcedureMemberRpcs.java:94) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:327) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) Caused by: org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:244) at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:238) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34) at org.mockito.internal.MockHandler.handle(MockHandler.java:99) at org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36) at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.acquireBarrier() at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:161) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl.call(Subprocedure.java:315) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) 2014-06-29 19:26:34,355 ERROR [pool-1-thread-1-EventThread] procedure.ProcedureMember(250): Propagating foreign exception to subprocedure op org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via TIMER:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.errorhandling.ForeignException.deserialize(ForeignException.java:171) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.abort(ZKProcedureMemberRpcs.java:320) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs$1.nodeCreated(ZKProcedureMemberRpcs.java:94) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:327) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) Caused by: org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:244) at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:238) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34) at org.mockito.internal.MockHandler.handle(MockHandler.java:99) at org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36) at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.acquireBarrier() at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:161) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl.call(Subprocedure.java:315) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) 2014-06-29 19:26:34,357 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility-0x146e917cf2b0008, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/error injection coordination/abort 2014-06-29 19:26:34,358 ERROR [pool-1-thread-1-EventThread] procedure.Subprocedure(265): Propagating foreign exception to subprocedure op org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via TIMER:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.errorhandling.ForeignException.deserialize(ForeignException.java:171) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.abort(ZKProcedureMemberRpcs.java:320) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs$1.nodeCreated(ZKProcedureMemberRpcs.java:94) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:327) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) Caused by: org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:244) at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:238) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34) at org.mockito.internal.MockHandler.handle(MockHandler.java:99) at org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36) at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.acquireBarrier() at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:161) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl.call(Subprocedure.java:315) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) 2014-06-29 19:26:34,357 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility-0x146e917cf2b0009, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/error injection coordination/abort 2014-06-29 19:26:34,358 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility-0x146e917cf2b000a, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/error injection coordination/abort 2014-06-29 19:26:34,358 INFO [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs$1(109): Received procedure abort children changed event: /hbase/error injection coordination/abort 2014-06-29 19:26:34,356 DEBUG [pool-1-thread-1-EventThread] procedure.Procedure(303): Waiting on: java.util.concurrent.CountDownLatch@13dd208[Count = 2] remaining members to acquire global barrier 2014-06-29 19:26:34,359 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/error injection coordination/abort' 2014-06-29 19:26:34,359 INFO [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs$1(109): Received procedure abort children changed event: /hbase/error injection coordination/abort 2014-06-29 19:26:34,359 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/error injection coordination/abort' 2014-06-29 19:26:34,358 INFO [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs$1(109): Received procedure abort children changed event: /hbase/error injection coordination/abort 2014-06-29 19:26:34,359 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/error injection coordination/abort' 2014-06-29 19:26:34,359 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility-0x146e917cf2b0007, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/error injection coordination/abort/op 2014-06-29 19:26:34,360 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(170): Node created: /hbase/error injection coordination/abort/op 2014-06-29 19:26:34,360 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2014-06-29 19:26:34,360 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/error injection coordination 2014-06-29 19:26:34,360 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(304): Aborting procedure member for znode /hbase/error injection coordination/abort/op 2014-06-29 19:26:34,360 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(304): Aborting procedure member for znode /hbase/error injection coordination/abort/op 2014-06-29 19:26:34,361 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2014-06-29 19:26:34,361 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(304): Aborting procedure member for znode /hbase/error injection coordination/abort/op 2014-06-29 19:26:34,362 DEBUG [pool-1-thread-1-EventThread] procedure.ProcedureMember(241): Request received to abort procedure op org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via TIMER:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.errorhandling.ForeignException.deserialize(ForeignException.java:171) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.abort(ZKProcedureMemberRpcs.java:320) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.watchForAbortedProcedures(ZKProcedureMemberRpcs.java:142) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.access$200(ZKProcedureMemberRpcs.java:55) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs$1.nodeChildrenChanged(ZKProcedureMemberRpcs.java:110) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:348) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) Caused by: org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:244) at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:238) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34) at org.mockito.internal.MockHandler.handle(MockHandler.java:99) at org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36) at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.acquireBarrier() at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:161) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl.call(Subprocedure.java:315) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) 2014-06-29 19:26:34,362 DEBUG [pool-1-thread-1-EventThread] procedure.ProcedureMember(241): Request received to abort procedure op org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via TIMER:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.errorhandling.ForeignException.deserialize(ForeignException.java:171) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.abort(ZKProcedureMemberRpcs.java:320) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.watchForAbortedProcedures(ZKProcedureMemberRpcs.java:142) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.access$200(ZKProcedureMemberRpcs.java:55) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs$1.nodeChildrenChanged(ZKProcedureMemberRpcs.java:110) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:348) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) Caused by: org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:244) at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:238) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34) at org.mockito.internal.MockHandler.handle(MockHandler.java:99) at org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36) at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.acquireBarrier() at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:161) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl.call(Subprocedure.java:315) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) 2014-06-29 19:26:34,363 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |----op 2014-06-29 19:26:34,363 ERROR [pool-1-thread-1-EventThread] procedure.ProcedureMember(250): Propagating foreign exception to subprocedure op org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via TIMER:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.errorhandling.ForeignException.deserialize(ForeignException.java:171) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.abort(ZKProcedureMemberRpcs.java:320) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.watchForAbortedProcedures(ZKProcedureMemberRpcs.java:142) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.access$200(ZKProcedureMemberRpcs.java:55) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs$1.nodeChildrenChanged(ZKProcedureMemberRpcs.java:110) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:348) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) Caused by: org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:244) at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:238) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34) at org.mockito.internal.MockHandler.handle(MockHandler.java:99) at org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36) at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.acquireBarrier() at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:161) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl.call(Subprocedure.java:315) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) 2014-06-29 19:26:34,362 DEBUG [pool-1-thread-1-EventThread] procedure.ProcedureMember(241): Request received to abort procedure op org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via TIMER:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.errorhandling.ForeignException.deserialize(ForeignException.java:171) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.abort(ZKProcedureMemberRpcs.java:320) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.watchForAbortedProcedures(ZKProcedureMemberRpcs.java:142) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.access$200(ZKProcedureMemberRpcs.java:55) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs$1.nodeChildrenChanged(ZKProcedureMemberRpcs.java:110) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:348) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) Caused by: org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:244) at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:238) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34) at org.mockito.internal.MockHandler.handle(MockHandler.java:99) at org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36) at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.acquireBarrier() at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:161) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl.call(Subprocedure.java:315) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) 2014-06-29 19:26:34,362 ERROR [pool-1-thread-1-EventThread] procedure.ProcedureMember(250): Propagating foreign exception to subprocedure op org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via TIMER:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.errorhandling.ForeignException.deserialize(ForeignException.java:171) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.abort(ZKProcedureMemberRpcs.java:320) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.watchForAbortedProcedures(ZKProcedureMemberRpcs.java:142) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.access$200(ZKProcedureMemberRpcs.java:55) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs$1.nodeChildrenChanged(ZKProcedureMemberRpcs.java:110) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:348) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) Caused by: org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:244) at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:238) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34) at org.mockito.internal.MockHandler.handle(MockHandler.java:99) at org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36) at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.acquireBarrier() at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:161) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl.call(Subprocedure.java:315) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) 2014-06-29 19:26:34,364 ERROR [pool-1-thread-1-EventThread] procedure.ProcedureMember(250): Propagating foreign exception to subprocedure op org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via TIMER:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.errorhandling.ForeignException.deserialize(ForeignException.java:171) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.abort(ZKProcedureMemberRpcs.java:320) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.watchForAbortedProcedures(ZKProcedureMemberRpcs.java:142) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.access$200(ZKProcedureMemberRpcs.java:55) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs$1.nodeChildrenChanged(ZKProcedureMemberRpcs.java:110) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:348) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) Caused by: org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:244) at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:238) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34) at org.mockito.internal.MockHandler.handle(MockHandler.java:99) at org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36) at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.acquireBarrier() at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:161) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl.call(Subprocedure.java:315) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) 2014-06-29 19:26:34,363 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-------one 2014-06-29 19:26:34,363 ERROR [pool-1-thread-1-EventThread] procedure.Subprocedure(265): Propagating foreign exception to subprocedure op org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via TIMER:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.errorhandling.ForeignException.deserialize(ForeignException.java:171) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.abort(ZKProcedureMemberRpcs.java:320) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.watchForAbortedProcedures(ZKProcedureMemberRpcs.java:142) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.access$200(ZKProcedureMemberRpcs.java:55) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs$1.nodeChildrenChanged(ZKProcedureMemberRpcs.java:110) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:348) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) Caused by: org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:244) at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:238) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34) at org.mockito.internal.MockHandler.handle(MockHandler.java:99) at org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36) at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.acquireBarrier() at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:161) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl.call(Subprocedure.java:315) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) 2014-06-29 19:26:34,365 ERROR [pool-1-thread-1-EventThread] procedure.Subprocedure(265): Propagating foreign exception to subprocedure op org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via TIMER:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.errorhandling.ForeignException.deserialize(ForeignException.java:171) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.abort(ZKProcedureMemberRpcs.java:320) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.watchForAbortedProcedures(ZKProcedureMemberRpcs.java:142) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.access$200(ZKProcedureMemberRpcs.java:55) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs$1.nodeChildrenChanged(ZKProcedureMemberRpcs.java:110) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:348) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) Caused by: org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:244) at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:238) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34) at org.mockito.internal.MockHandler.handle(MockHandler.java:99) at org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36) at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.acquireBarrier() at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:161) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl.call(Subprocedure.java:315) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) 2014-06-29 19:26:34,364 ERROR [pool-1-thread-1-EventThread] procedure.Subprocedure(265): Propagating foreign exception to subprocedure op org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via TIMER:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.errorhandling.ForeignException.deserialize(ForeignException.java:171) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.abort(ZKProcedureMemberRpcs.java:320) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.watchForAbortedProcedures(ZKProcedureMemberRpcs.java:142) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.access$200(ZKProcedureMemberRpcs.java:55) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs$1.nodeChildrenChanged(ZKProcedureMemberRpcs.java:110) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:348) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) Caused by: org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:244) at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:238) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34) at org.mockito.internal.MockHandler.handle(MockHandler.java:99) at org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36) at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.acquireBarrier() at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:161) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl.call(Subprocedure.java:315) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) 2014-06-29 19:26:34,365 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-abort 2014-06-29 19:26:34,367 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |----op 2014-06-29 19:26:34,368 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-reached 2014-06-29 19:26:34,370 DEBUG [pool-1-thread-1-EventThread] errorhandling.ForeignExceptionDispatcher(66): accepting received exception org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via TIMER:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.errorhandling.ForeignException.deserialize(ForeignException.java:171) at org.apache.hadoop.hbase.procedure.ZKProcedureCoordinatorRpcs.abort(ZKProcedureCoordinatorRpcs.java:243) at org.apache.hadoop.hbase.procedure.ZKProcedureCoordinatorRpcs$1.nodeCreated(ZKProcedureCoordinatorRpcs.java:183) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:327) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) Caused by: org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:244) at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:238) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34) at org.mockito.internal.MockHandler.handle(MockHandler.java:99) at org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36) at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.acquireBarrier() at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:161) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl.call(Subprocedure.java:315) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) 2014-06-29 19:26:34,370 DEBUG [member: 'two' subprocedure-pool10-thread-1] procedure.Subprocedure(162): Subprocedure 'op' locally acquired 2014-06-29 19:26:34,371 DEBUG [member: 'two' subprocedure-pool10-thread-1] procedure.ZKProcedureMemberRpcs(237): Member: 'two' joining acquired barrier for procedure (op) in zk 2014-06-29 19:26:34,392 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility-0x146e917cf2b0007, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/error injection coordination/acquired/op/two 2014-06-29 19:26:34,392 DEBUG [member: 'two' subprocedure-pool10-thread-1] procedure.ZKProcedureMemberRpcs(245): Watch for global barrier reached:/hbase/error injection coordination/reached/op 2014-06-29 19:26:34,392 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(170): Node created: /hbase/error injection coordination/acquired/op/two 2014-06-29 19:26:34,393 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2014-06-29 19:26:34,393 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/error injection coordination 2014-06-29 19:26:34,393 DEBUG [member: 'two' subprocedure-pool10-thread-1] zookeeper.ZKUtil(430): testing utility-0x146e917cf2b0009, quorum=localhost:51439, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/error injection coordination/reached/op 2014-06-29 19:26:34,393 DEBUG [member: 'two' subprocedure-pool10-thread-1] procedure.Subprocedure(166): Subprocedure 'op' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2014-06-29 19:26:34,393 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2014-06-29 19:26:34,394 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |----op 2014-06-29 19:26:34,395 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-------two 2014-06-29 19:26:34,395 ERROR [member: 'two' subprocedure-pool10-thread-1] procedure.Subprocedure(265): Subprocedure 'op' aborting due to a ForeignException! org.apache.hadoop.hbase.errorhandling.TimeoutException via TIMER:org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.errorhandling.ForeignExceptionDispatcher.rethrowException(ForeignExceptionDispatcher.java:83) at org.apache.hadoop.hbase.procedure.Procedure.waitForLatch(Procedure.java:369) at org.apache.hadoop.hbase.procedure.Subprocedure.waitForReachedGlobalBarrier(Subprocedure.java:296) at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:170) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl.call(Subprocedure.java:315) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) Caused by: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:244) at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:238) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34) at org.mockito.internal.MockHandler.handle(MockHandler.java:99) at org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36) at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.acquireBarrier() at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:161) ... 6 more 2014-06-29 19:26:34,395 DEBUG [member: 'two' subprocedure-pool10-thread-1] procedure.Subprocedure(203): Subprocedure 'op' running cleanup. 2014-06-29 19:26:34,395 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-------one 2014-06-29 19:26:34,396 DEBUG [member: 'two' subprocedure-pool10-thread-1] errorhandling.TimeoutExceptionInjector(89): Marking timer as complete - no error notifications will be received for this timer. 2014-06-29 19:26:34,396 DEBUG [member: 'two' subprocedure-pool10-thread-1] procedure.Subprocedure(211): Subprocedure 'op' completed. 2014-06-29 19:26:34,396 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-abort 2014-06-29 19:26:34,397 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |----op 2014-06-29 19:26:34,397 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-reached 2014-06-29 19:26:34,398 DEBUG [pool-1-thread-1-EventThread] procedure.Procedure(294): member: 'two' joining acquired barrier for procedure 'op' on coordinator 2014-06-29 19:26:34,398 DEBUG [pool-1-thread-1-EventThread] procedure.Procedure(303): Waiting on: java.util.concurrent.CountDownLatch@13dd208[Count = 1] remaining members to acquire global barrier 2014-06-29 19:26:34,579 DEBUG [pool-1-thread-1] procedure.TestZKProcedure(392): Checking mock:0 2014-06-29 19:26:34,598 ERROR [(coordinator)-proc-coordinator-pool8-thread-1] procedure.Procedure(223): Procedure 'op' execution failed! org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via TIMER:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.errorhandling.ForeignExceptionDispatcher.rethrowException(ForeignExceptionDispatcher.java:83) at org.apache.hadoop.hbase.procedure.Procedure.waitForLatch(Procedure.java:369) at org.apache.hadoop.hbase.procedure.Procedure.call(Procedure.java:206) at org.apache.hadoop.hbase.procedure.Procedure.call(Procedure.java:67) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) Caused by: org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:244) at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:238) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34) at org.mockito.internal.MockHandler.handle(MockHandler.java:99) at org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36) at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.acquireBarrier() at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:161) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl.call(Subprocedure.java:315) ... 5 more 2014-06-29 19:26:34,598 DEBUG [(coordinator)-proc-coordinator-pool8-thread-1] procedure.Procedure(226): Running finish phase. 2014-06-29 19:26:34,599 DEBUG [(coordinator)-proc-coordinator-pool8-thread-1] procedure.Procedure(277): Finished coordinator procedure - removing self from list of running procedures 2014-06-29 19:26:34,599 DEBUG [(coordinator)-proc-coordinator-pool8-thread-1] procedure.ZKProcedureCoordinatorRpcs(141): Attempting to clean out zk node for op:op 2014-06-29 19:26:34,599 INFO [(coordinator)-proc-coordinator-pool8-thread-1] procedure.ZKProcedureUtil(284): Clearing all znodes for procedure opincluding nodes /hbase/error injection coordination/acquired /hbase/error injection coordination/reached /hbase/error injection coordination/abort 2014-06-29 19:26:34,603 INFO [pool-1-thread-1] hbase.ResourceChecker(171): after: procedure.TestZKProcedure#testMultiCohortWithMemberTimeoutDuringPrepare Thread=38 (was 24) Potentially hanging thread: Timer-9 java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:485) java.util.TimerThread.mainLoop(Timer.java:483) java.util.TimerThread.run(Timer.java:462) Potentially hanging thread: member: 'two' subprocedure-pool10-thread-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: member: 'three' subprocedure-pool11-thread-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1011) java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1303) java.util.concurrent.CountDownLatch.await(CountDownLatch.java:253) org.apache.hadoop.hbase.procedure.Procedure.waitForLatch(Procedure.java:374) org.apache.hadoop.hbase.procedure.Subprocedure.waitForReachedGlobalBarrier(Subprocedure.java:296) org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.CGLIB$waitForReachedGlobalBarrier$11() org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547$$FastClassByMockitoWithCGLIB$$cd77604e.invoke() org.mockito.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:216) org.mockito.internal.creation.AbstractMockitoMethodProxy.invokeSuper(AbstractMockitoMethodProxy.java:10) org.mockito.internal.invocation.realmethod.CGLIBProxyRealMethod.invoke(CGLIBProxyRealMethod.java:22) org.mockito.internal.invocation.realmethod.FilteredCGLIBProxyRealMethod.invoke(FilteredCGLIBProxyRealMethod.java:27) org.mockito.internal.invocation.Invocation.callRealMethod(Invocation.java:213) org.mockito.internal.stubbing.answers.CallsRealMethods.answer(CallsRealMethods.java:36) org.mockito.internal.MockHandler.handle(MockHandler.java:101) org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36) org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48) org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.waitForReachedGlobalBarrier() org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:170) org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl.call(Subprocedure.java:315) java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) java.util.concurrent.FutureTask.run(FutureTask.java:138) java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: member: 'one' subprocedure-pool9-thread-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1011) java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1303) java.util.concurrent.CountDownLatch.await(CountDownLatch.java:253) org.apache.hadoop.hbase.procedure.Procedure.waitForLatch(Procedure.java:374) org.apache.hadoop.hbase.procedure.Subprocedure.waitForReachedGlobalBarrier(Subprocedure.java:296) org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.CGLIB$waitForReachedGlobalBarrier$11() org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547$$FastClassByMockitoWithCGLIB$$cd77604e.invoke() org.mockito.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:216) org.mockito.internal.creation.AbstractMockitoMethodProxy.invokeSuper(AbstractMockitoMethodProxy.java:10) org.mockito.internal.invocation.realmethod.CGLIBProxyRealMethod.invoke(CGLIBProxyRealMethod.java:22) org.mockito.internal.invocation.realmethod.FilteredCGLIBProxyRealMethod.invoke(FilteredCGLIBProxyRealMethod.java:27) org.mockito.internal.invocation.Invocation.callRealMethod(Invocation.java:213) org.mockito.internal.stubbing.answers.CallsRealMethods.answer(CallsRealMethods.java:36) org.mockito.internal.MockHandler.handle(MockHandler.java:101) org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36) org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48) org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.waitForReachedGlobalBarrier() org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:170) org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl.call(Subprocedure.java:315) java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) java.util.concurrent.FutureTask.run(FutureTask.java:138) java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: (coordinator)-proc-coordinator-pool8-thread-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1011) java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1303) java.util.concurrent.CountDownLatch.await(CountDownLatch.java:253) org.apache.hadoop.hbase.procedure.Procedure.waitForLatch(Procedure.java:374) org.apache.hadoop.hbase.procedure.Procedure.call(Procedure.java:206) org.apache.hadoop.hbase.procedure.Procedure.call(Procedure.java:67) java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) java.util.concurrent.FutureTask.run(FutureTask.java:138) java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: Timer-7 java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:509) java.util.TimerThread.run(Timer.java:462) Potentially hanging thread: Timer-10 java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:509) java.util.TimerThread.run(Timer.java:462) - Thread LEAK? -, OpenFileDescriptor=187 (was 173) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=40000 (was 40000), SystemLoadAverage=360 (was 360), ProcessCount=168 (was 168), AvailableMemoryMB=3166 (was 3168), ConnectionCount=0 (was 0) 2014-06-29 19:26:34,631 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility-0x146e917cf2b0008, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/error injection coordination/acquired 2014-06-29 19:26:34,632 INFO [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs$1(106): Received procedure start children changed event: /hbase/error injection coordination/acquired 2014-06-29 19:26:34,632 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility-0x146e917cf2b0009, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/error injection coordination/acquired 2014-06-29 19:26:34,632 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility-0x146e917cf2b000a, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/error injection coordination/acquired 2014-06-29 19:26:34,632 INFO [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs$1(106): Received procedure start children changed event: /hbase/error injection coordination/acquired 2014-06-29 19:26:34,632 INFO [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs$1(106): Received procedure start children changed event: /hbase/error injection coordination/acquired 2014-06-29 19:26:34,632 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/error injection coordination/acquired' 2014-06-29 19:26:34,633 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/error injection coordination/acquired' 2014-06-29 19:26:34,632 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/error injection coordination/acquired' 2014-06-29 19:26:34,638 ERROR [member: 'three' subprocedure-pool11-thread-1] procedure.Subprocedure(265): Subprocedure 'op' aborting due to a ForeignException! org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via TIMER:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.errorhandling.ForeignExceptionDispatcher.rethrowException(ForeignExceptionDispatcher.java:83) at org.apache.hadoop.hbase.procedure.Procedure.waitForLatch(Procedure.java:369) at org.apache.hadoop.hbase.procedure.Subprocedure.waitForReachedGlobalBarrier(Subprocedure.java:296) at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:170) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl.call(Subprocedure.java:315) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) Caused by: org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:244) at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:238) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34) at org.mockito.internal.MockHandler.handle(MockHandler.java:99) at org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36) at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.acquireBarrier() at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:161) ... 6 more 2014-06-29 19:26:34,638 DEBUG [member: 'three' subprocedure-pool11-thread-1] procedure.Subprocedure(203): Subprocedure 'op' running cleanup. 2014-06-29 19:26:34,638 DEBUG [member: 'three' subprocedure-pool11-thread-1] errorhandling.TimeoutExceptionInjector(89): Marking timer as complete - no error notifications will be received for this timer. 2014-06-29 19:26:34,638 DEBUG [member: 'three' subprocedure-pool11-thread-1] procedure.Subprocedure(211): Subprocedure 'op' completed. 2014-06-29 19:26:34,639 ERROR [member: 'one' subprocedure-pool9-thread-1] procedure.Subprocedure(265): Subprocedure 'op' aborting due to a ForeignException! org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via TIMER:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.errorhandling.ForeignExceptionDispatcher.rethrowException(ForeignExceptionDispatcher.java:83) at org.apache.hadoop.hbase.procedure.Procedure.waitForLatch(Procedure.java:369) at org.apache.hadoop.hbase.procedure.Subprocedure.waitForReachedGlobalBarrier(Subprocedure.java:296) at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:170) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl.call(Subprocedure.java:315) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) Caused by: org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:244) at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:238) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34) at org.mockito.internal.MockHandler.handle(MockHandler.java:99) at org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36) at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.acquireBarrier() at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:161) ... 6 more 2014-06-29 19:26:34,639 DEBUG [member: 'one' subprocedure-pool9-thread-1] procedure.Subprocedure(203): Subprocedure 'op' running cleanup. 2014-06-29 19:26:34,639 WARN [member: 'one' subprocedure-pool9-thread-1] errorhandling.TimeoutExceptionInjector(86): Timer already marked completed, ignoring! 2014-06-29 19:26:34,639 DEBUG [member: 'one' subprocedure-pool9-thread-1] procedure.Subprocedure(211): Subprocedure 'op' completed. 2014-06-29 19:26:34,664 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility-0x146e917cf2b000a, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/error injection coordination/abort 2014-06-29 19:26:34,665 DEBUG [(coordinator)-proc-coordinator-pool8-thread-1] errorhandling.TimeoutExceptionInjector(89): Marking timer as complete - no error notifications will be received for this timer. 2014-06-29 19:26:34,665 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility-0x146e917cf2b0008, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/error injection coordination/abort 2014-06-29 19:26:34,665 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility-0x146e917cf2b0009, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/error injection coordination/abort 2014-06-29 19:26:34,665 INFO [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs$1(109): Received procedure abort children changed event: /hbase/error injection coordination/abort 2014-06-29 19:26:34,666 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/error injection coordination/abort' 2014-06-29 19:26:34,665 INFO [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs$1(109): Received procedure abort children changed event: /hbase/error injection coordination/abort 2014-06-29 19:26:34,665 INFO [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs$1(109): Received procedure abort children changed event: /hbase/error injection coordination/abort 2014-06-29 19:26:34,666 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/error injection coordination/abort' 2014-06-29 19:26:34,666 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/error injection coordination/abort' 2014-06-29 19:26:34,619 INFO [pool-1-thread-1] hbase.ResourceChecker(147): before: procedure.TestZKProcedure#testEmptyMemberSet Thread=38, OpenFileDescriptor=189, MaxFileDescriptor=40000, SystemLoadAverage=360, ProcessCount=168, AvailableMemoryMB=3166, ConnectionCount=0 2014-06-29 19:26:34,621 INFO [pool-1-thread-1] zookeeper.RecoverableZooKeeper(120): Process identifier=testing utility connecting to ZooKeeper ensemble=localhost:51439 2014-06-29 19:26:34,643 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2014-06-29 19:26:34,644 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(387): testing utility-0x146e917cf2b000b connected 2014-06-29 19:26:34,709 INFO [pool-1-thread-1] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/coordination test - 0 cohort members/acquired /hbase/coordination test - 0 cohort members/reached /hbase/coordination test - 0 cohort members/abort 2014-06-29 19:26:34,711 DEBUG [pool-1-thread-1] procedure.ZKProcedureCoordinatorRpcs(195): Starting the controller for procedure member:coordinator 2014-06-29 19:26:34,712 INFO [(coordinator)-proc-coordinator-pool12-thread-1] procedure.Procedure(193): Starting procedure 'op' 2014-06-29 19:26:34,712 DEBUG [(coordinator)-proc-coordinator-pool12-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 60000 ms 2014-06-29 19:26:34,712 DEBUG [(coordinator)-proc-coordinator-pool12-thread-1] procedure.Procedure(201): Procedure 'op' starting 'acquire' 2014-06-29 19:26:34,713 DEBUG [(coordinator)-proc-coordinator-pool12-thread-1] procedure.Procedure(243): Starting procedure 'op', kicking off acquire phase on members. 2014-06-29 19:26:34,714 DEBUG [(coordinator)-proc-coordinator-pool12-thread-1] zookeeper.ZKUtil(430): testing utility-0x146e917cf2b000b, quorum=localhost:51439, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/coordination test - 0 cohort members/abort/op 2014-06-29 19:26:34,714 DEBUG [(coordinator)-proc-coordinator-pool12-thread-1] procedure.ZKProcedureCoordinatorRpcs(93): Creating acquire znode:/hbase/coordination test - 0 cohort members/acquired/op 2014-06-29 19:26:34,720 DEBUG [(coordinator)-proc-coordinator-pool12-thread-1] procedure.Procedure(205): Waiting for all members to 'acquire' 2014-06-29 19:26:34,720 DEBUG [(coordinator)-proc-coordinator-pool12-thread-1] procedure.Procedure(209): Procedure 'op' starting 'in-barrier' execution. 2014-06-29 19:26:34,720 DEBUG [(coordinator)-proc-coordinator-pool12-thread-1] procedure.ZKProcedureCoordinatorRpcs(115): Creating reached barrier zk node:/hbase/coordination test - 0 cohort members/reached/op 2014-06-29 19:26:34,731 INFO [(coordinator)-proc-coordinator-pool12-thread-1] procedure.Procedure(217): Procedure 'op' execution completed 2014-06-29 19:26:34,731 DEBUG [(coordinator)-proc-coordinator-pool12-thread-1] procedure.Procedure(226): Running finish phase. 2014-06-29 19:26:34,731 DEBUG [(coordinator)-proc-coordinator-pool12-thread-1] procedure.Procedure(277): Finished coordinator procedure - removing self from list of running procedures 2014-06-29 19:26:34,732 DEBUG [(coordinator)-proc-coordinator-pool12-thread-1] procedure.ZKProcedureCoordinatorRpcs(141): Attempting to clean out zk node for op:op 2014-06-29 19:26:34,732 INFO [(coordinator)-proc-coordinator-pool12-thread-1] procedure.ZKProcedureUtil(284): Clearing all znodes for procedure opincluding nodes /hbase/coordination test - 0 cohort members/acquired /hbase/coordination test - 0 cohort members/reached /hbase/coordination test - 0 cohort members/abort 2014-06-29 19:26:34,754 DEBUG [(coordinator)-proc-coordinator-pool12-thread-1] errorhandling.TimeoutExceptionInjector(89): Marking timer as complete - no error notifications will be received for this timer. 2014-06-29 19:26:34,770 INFO [pool-1-thread-1] hbase.ResourceChecker(171): after: procedure.TestZKProcedure#testEmptyMemberSet Thread=38 (was 38), OpenFileDescriptor=196 (was 189) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=40000 (was 40000), SystemLoadAverage=360 (was 360), ProcessCount=168 (was 168), AvailableMemoryMB=3166 (was 3166), ConnectionCount=0 (was 0) 2014-06-29 19:26:34,775 INFO [pool-1-thread-1] zookeeper.MiniZooKeeperCluster(246): Shutdown MiniZK cluster with all ZK servers Standard Error 2014-06-29 19:26:33,781 INFO [pool-1-thread-1] hbase.ResourceChecker(147): before: procedure.TestZKProcedure#testMultiCohortWithMemberTimeoutDuringPrepare Thread=24, OpenFileDescriptor=173, MaxFileDescriptor=40000, SystemLoadAverage=360, ProcessCount=168, AvailableMemoryMB=3168, ConnectionCount=0 2014-06-29 19:26:33,785 INFO [pool-1-thread-1] zookeeper.RecoverableZooKeeper(120): Process identifier=testing utility connecting to ZooKeeper ensemble=localhost:51439 2014-06-29 19:26:33,799 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2014-06-29 19:26:33,801 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(387): testing utility-0x146e917cf2b0007 connected 2014-06-29 19:26:33,882 INFO [pool-1-thread-1] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/error injection coordination/acquired /hbase/error injection coordination/reached /hbase/error injection coordination/abort 2014-06-29 19:26:33,884 DEBUG [pool-1-thread-1] procedure.ZKProcedureCoordinatorRpcs(195): Starting the controller for procedure member:coordinator 2014-06-29 19:26:33,897 INFO [pool-1-thread-1] zookeeper.RecoverableZooKeeper(120): Process identifier=testing utility connecting to ZooKeeper ensemble=localhost:51439 2014-06-29 19:26:33,914 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2014-06-29 19:26:33,916 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(387): testing utility-0x146e917cf2b0008 connected 2014-06-29 19:26:33,936 INFO [pool-1-thread-1] zookeeper.RecoverableZooKeeper(508): Node /hbase/error injection coordination/acquired already exists and this is not a retry 2014-06-29 19:26:33,938 DEBUG [pool-1-thread-1] procedure.ZKProcedureMemberRpcs(337): Starting procedure member 'one' 2014-06-29 19:26:33,938 DEBUG [pool-1-thread-1] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/error injection coordination/abort' 2014-06-29 19:26:33,939 DEBUG [pool-1-thread-1] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/error injection coordination/acquired' 2014-06-29 19:26:33,942 INFO [pool-1-thread-1] zookeeper.RecoverableZooKeeper(120): Process identifier=testing utility connecting to ZooKeeper ensemble=localhost:51439 2014-06-29 19:26:33,959 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2014-06-29 19:26:33,960 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(387): testing utility-0x146e917cf2b0009 connected 2014-06-29 19:26:33,980 INFO [pool-1-thread-1] zookeeper.RecoverableZooKeeper(508): Node /hbase/error injection coordination/acquired already exists and this is not a retry 2014-06-29 19:26:33,982 DEBUG [pool-1-thread-1] procedure.ZKProcedureMemberRpcs(337): Starting procedure member 'two' 2014-06-29 19:26:33,982 DEBUG [pool-1-thread-1] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/error injection coordination/abort' 2014-06-29 19:26:33,983 DEBUG [pool-1-thread-1] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/error injection coordination/acquired' 2014-06-29 19:26:33,986 INFO [pool-1-thread-1] zookeeper.RecoverableZooKeeper(120): Process identifier=testing utility connecting to ZooKeeper ensemble=localhost:51439 2014-06-29 19:26:34,004 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2014-06-29 19:26:34,005 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(387): testing utility-0x146e917cf2b000a connected 2014-06-29 19:26:34,034 INFO [pool-1-thread-1] zookeeper.RecoverableZooKeeper(508): Node /hbase/error injection coordination/acquired already exists and this is not a retry 2014-06-29 19:26:34,036 DEBUG [pool-1-thread-1] procedure.ZKProcedureMemberRpcs(337): Starting procedure member 'three' 2014-06-29 19:26:34,036 DEBUG [pool-1-thread-1] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/error injection coordination/abort' 2014-06-29 19:26:34,037 DEBUG [pool-1-thread-1] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/error injection coordination/acquired' 2014-06-29 19:26:34,065 INFO [(coordinator)-proc-coordinator-pool8-thread-1] procedure.Procedure(193): Starting procedure 'op' 2014-06-29 19:26:34,065 DEBUG [(coordinator)-proc-coordinator-pool8-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 10000 ms 2014-06-29 19:26:34,077 DEBUG [(coordinator)-proc-coordinator-pool8-thread-1] procedure.Procedure(201): Procedure 'op' starting 'acquire' 2014-06-29 19:26:34,077 DEBUG [(coordinator)-proc-coordinator-pool8-thread-1] procedure.Procedure(243): Starting procedure 'op', kicking off acquire phase on members. 2014-06-29 19:26:34,078 DEBUG [(coordinator)-proc-coordinator-pool8-thread-1] zookeeper.ZKUtil(430): testing utility-0x146e917cf2b0007, quorum=localhost:51439, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/error injection coordination/abort/op 2014-06-29 19:26:34,078 DEBUG [(coordinator)-proc-coordinator-pool8-thread-1] procedure.ZKProcedureCoordinatorRpcs(93): Creating acquire znode:/hbase/error injection coordination/acquired/op 2014-06-29 19:26:34,092 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility-0x146e917cf2b000a, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/error injection coordination/acquired 2014-06-29 19:26:34,092 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility-0x146e917cf2b0009, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/error injection coordination/acquired 2014-06-29 19:26:34,092 DEBUG [(coordinator)-proc-coordinator-pool8-thread-1] procedure.ZKProcedureCoordinatorRpcs(101): Watching for acquire node:/hbase/error injection coordination/acquired/op/one 2014-06-29 19:26:34,092 INFO [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs$1(106): Received procedure start children changed event: /hbase/error injection coordination/acquired 2014-06-29 19:26:34,092 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/error injection coordination/acquired' 2014-06-29 19:26:34,092 INFO [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs$1(106): Received procedure start children changed event: /hbase/error injection coordination/acquired 2014-06-29 19:26:34,092 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility-0x146e917cf2b0008, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/error injection coordination/acquired 2014-06-29 19:26:34,093 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/error injection coordination/acquired' 2014-06-29 19:26:34,093 INFO [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs$1(106): Received procedure start children changed event: /hbase/error injection coordination/acquired 2014-06-29 19:26:34,093 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/error injection coordination/acquired' 2014-06-29 19:26:34,094 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(184): Found procedure znode: /hbase/error injection coordination/acquired/op 2014-06-29 19:26:34,094 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(184): Found procedure znode: /hbase/error injection coordination/acquired/op 2014-06-29 19:26:34,094 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(184): Found procedure znode: /hbase/error injection coordination/acquired/op 2014-06-29 19:26:34,094 DEBUG [(coordinator)-proc-coordinator-pool8-thread-1] zookeeper.ZKUtil(430): testing utility-0x146e917cf2b0007, quorum=localhost:51439, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/error injection coordination/acquired/op/one 2014-06-29 19:26:34,094 DEBUG [(coordinator)-proc-coordinator-pool8-thread-1] procedure.ZKProcedureCoordinatorRpcs(101): Watching for acquire node:/hbase/error injection coordination/acquired/op/two 2014-06-29 19:26:34,095 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(430): testing utility-0x146e917cf2b000a, quorum=localhost:51439, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/error injection coordination/abort/op 2014-06-29 19:26:34,095 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(430): testing utility-0x146e917cf2b0009, quorum=localhost:51439, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/error injection coordination/abort/op 2014-06-29 19:26:34,095 DEBUG [(coordinator)-proc-coordinator-pool8-thread-1] zookeeper.ZKUtil(430): testing utility-0x146e917cf2b0007, quorum=localhost:51439, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/error injection coordination/acquired/op/two 2014-06-29 19:26:34,095 DEBUG [(coordinator)-proc-coordinator-pool8-thread-1] procedure.ZKProcedureCoordinatorRpcs(101): Watching for acquire node:/hbase/error injection coordination/acquired/op/three 2014-06-29 19:26:34,095 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(430): testing utility-0x146e917cf2b0008, quorum=localhost:51439, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/error injection coordination/abort/op 2014-06-29 19:26:34,096 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(203): start proc data length is 6 2014-06-29 19:26:34,096 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(203): start proc data length is 6 2014-06-29 19:26:34,096 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(212): Found data for znode:/hbase/error injection coordination/acquired/op 2014-06-29 19:26:34,096 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(212): Found data for znode:/hbase/error injection coordination/acquired/op 2014-06-29 19:26:34,096 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(203): start proc data length is 6 2014-06-29 19:26:34,096 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(212): Found data for znode:/hbase/error injection coordination/acquired/op 2014-06-29 19:26:34,097 DEBUG [(coordinator)-proc-coordinator-pool8-thread-1] zookeeper.ZKUtil(430): testing utility-0x146e917cf2b0007, quorum=localhost:51439, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/error injection coordination/acquired/op/three 2014-06-29 19:26:34,097 DEBUG [(coordinator)-proc-coordinator-pool8-thread-1] procedure.Procedure(205): Waiting for all members to 'acquire' 2014-06-29 19:26:34,098 DEBUG [pool-1-thread-1-EventThread] procedure.ProcedureMember(155): Submitting new Subprocedure:op 2014-06-29 19:26:34,098 DEBUG [pool-1-thread-1-EventThread] procedure.ProcedureMember(155): Submitting new Subprocedure:op 2014-06-29 19:26:34,098 DEBUG [member: 'three' subprocedure-pool11-thread-1] procedure.Subprocedure(152): Starting subprocedure 'op' with timeout 10000ms 2014-06-29 19:26:34,098 DEBUG [pool-1-thread-1-EventThread] procedure.ProcedureMember(155): Submitting new Subprocedure:op 2014-06-29 19:26:34,099 DEBUG [member: 'three' subprocedure-pool11-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 10000 ms 2014-06-29 19:26:34,099 DEBUG [member: 'one' subprocedure-pool9-thread-1] procedure.Subprocedure(152): Starting subprocedure 'op' with timeout 10000ms 2014-06-29 19:26:34,099 DEBUG [member: 'three' subprocedure-pool11-thread-1] procedure.Subprocedure(160): Subprocedure 'op' starting 'acquire' stage 2014-06-29 19:26:34,099 WARN [member: 'one' subprocedure-pool9-thread-1] errorhandling.TimeoutExceptionInjector(105): Timer already started, can't be started again. Ignoring second request. 2014-06-29 19:26:34,100 DEBUG [member: 'one' subprocedure-pool9-thread-1] procedure.Subprocedure(160): Subprocedure 'op' starting 'acquire' stage 2014-06-29 19:26:34,099 DEBUG [member: 'two' subprocedure-pool10-thread-1] procedure.Subprocedure(152): Starting subprocedure 'op' with timeout 10000ms 2014-06-29 19:26:34,100 DEBUG [member: 'two' subprocedure-pool10-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 10000 ms 2014-06-29 19:26:34,100 DEBUG [member: 'two' subprocedure-pool10-thread-1] procedure.Subprocedure(160): Subprocedure 'op' starting 'acquire' stage 2014-06-29 19:26:34,100 DEBUG [member: 'three' subprocedure-pool11-thread-1] procedure.Subprocedure(162): Subprocedure 'op' locally acquired 2014-06-29 19:26:34,101 DEBUG [member: 'three' subprocedure-pool11-thread-1] procedure.ZKProcedureMemberRpcs(237): Member: 'one' joining acquired barrier for procedure (op) in zk 2014-06-29 19:26:34,101 DEBUG [member: 'one' subprocedure-pool9-thread-1] procedure.Subprocedure(162): Subprocedure 'op' locally acquired 2014-06-29 19:26:34,101 DEBUG [member: 'one' subprocedure-pool9-thread-1] procedure.ZKProcedureMemberRpcs(237): Member: 'one' joining acquired barrier for procedure (op) in zk 2014-06-29 19:26:34,102 DEBUG [member: 'two' subprocedure-pool10-thread-1] procedure.TestZKProcedure$4(243): Sending error to coordinator 2014-06-29 19:26:34,103 ERROR [member: 'two' subprocedure-pool10-thread-1] procedure.TestZKProcedure$4(247): Remote commit failure, not propagating error:org.apache.hadoop.hbase.errorhandling.TimeoutException via TIMER:org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms 2014-06-29 19:26:34,103 DEBUG [member: 'two' subprocedure-pool10-thread-1] procedure.ProcedureMember(241): Request received to abort procedure op org.apache.hadoop.hbase.errorhandling.TimeoutException via TIMER:org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:244) at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:238) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34) at org.mockito.internal.MockHandler.handle(MockHandler.java:99) at org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36) at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.acquireBarrier() at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:161) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl.call(Subprocedure.java:315) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) Caused by: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms ... 14 more 2014-06-29 19:26:34,105 ERROR [member: 'two' subprocedure-pool10-thread-1] procedure.ProcedureMember(250): Propagating foreign exception to subprocedure op org.apache.hadoop.hbase.errorhandling.TimeoutException via TIMER:org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:244) at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:238) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34) at org.mockito.internal.MockHandler.handle(MockHandler.java:99) at org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36) at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.acquireBarrier() at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:161) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl.call(Subprocedure.java:315) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) Caused by: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms ... 14 more 2014-06-29 19:26:34,106 ERROR [member: 'two' subprocedure-pool10-thread-1] procedure.Subprocedure(265): Propagating foreign exception to subprocedure op org.apache.hadoop.hbase.errorhandling.TimeoutException via TIMER:org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:244) at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:238) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34) at org.mockito.internal.MockHandler.handle(MockHandler.java:99) at org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36) at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.acquireBarrier() at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:161) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl.call(Subprocedure.java:315) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) Caused by: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms ... 14 more 2014-06-29 19:26:34,107 DEBUG [member: 'two' subprocedure-pool10-thread-1] errorhandling.ForeignExceptionDispatcher(66): accepting received exception org.apache.hadoop.hbase.errorhandling.TimeoutException via TIMER:org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:244) at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:238) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34) at org.mockito.internal.MockHandler.handle(MockHandler.java:99) at org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36) at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.acquireBarrier() at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:161) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl.call(Subprocedure.java:315) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) Caused by: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms ... 14 more 2014-06-29 19:26:34,107 DEBUG [member: 'two' subprocedure-pool10-thread-1] procedure.ZKProcedureMemberRpcs(283): Aborting procedure (op) in zk 2014-06-29 19:26:34,120 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility-0x146e917cf2b0007, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/error injection coordination/acquired/op/one 2014-06-29 19:26:34,121 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(170): Node created: /hbase/error injection coordination/acquired/op/one 2014-06-29 19:26:34,121 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2014-06-29 19:26:34,122 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/error injection coordination 2014-06-29 19:26:34,122 DEBUG [member: 'three' subprocedure-pool11-thread-1] procedure.ZKProcedureMemberRpcs(245): Watch for global barrier reached:/hbase/error injection coordination/reached/op 2014-06-29 19:26:34,137 INFO [member: 'one' subprocedure-pool9-thread-1] zookeeper.RecoverableZooKeeper(508): Node /hbase/error injection coordination/acquired/op/one already exists and this is not a retry 2014-06-29 19:26:34,137 DEBUG [member: 'three' subprocedure-pool11-thread-1] zookeeper.ZKUtil(430): testing utility-0x146e917cf2b0008, quorum=localhost:51439, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/error injection coordination/reached/op 2014-06-29 19:26:34,137 DEBUG [member: 'three' subprocedure-pool11-thread-1] procedure.Subprocedure(166): Subprocedure 'op' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2014-06-29 19:26:34,137 DEBUG [member: 'one' subprocedure-pool9-thread-1] procedure.ZKProcedureMemberRpcs(245): Watch for global barrier reached:/hbase/error injection coordination/reached/op 2014-06-29 19:26:34,137 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2014-06-29 19:26:34,138 DEBUG [member: 'one' subprocedure-pool9-thread-1] zookeeper.ZKUtil(430): testing utility-0x146e917cf2b0008, quorum=localhost:51439, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/error injection coordination/reached/op 2014-06-29 19:26:34,138 DEBUG [member: 'one' subprocedure-pool9-thread-1] procedure.Subprocedure(166): Subprocedure 'op' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2014-06-29 19:26:34,343 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility-0x146e917cf2b000a, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/error injection coordination/abort/op 2014-06-29 19:26:34,344 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility-0x146e917cf2b0009, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/error injection coordination/abort/op 2014-06-29 19:26:34,344 DEBUG [member: 'two' subprocedure-pool10-thread-1] procedure.ZKProcedureMemberRpcs(289): Finished creating abort znode:/hbase/error injection coordination/abort/op 2014-06-29 19:26:34,344 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |----op 2014-06-29 19:26:34,344 INFO [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs$1(79): Received created event:/hbase/error injection coordination/abort/op 2014-06-29 19:26:34,344 INFO [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs$1(79): Received created event:/hbase/error injection coordination/abort/op 2014-06-29 19:26:34,344 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(304): Aborting procedure member for znode /hbase/error injection coordination/abort/op 2014-06-29 19:26:34,345 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(304): Aborting procedure member for znode /hbase/error injection coordination/abort/op 2014-06-29 19:26:34,344 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility-0x146e917cf2b0008, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/error injection coordination/abort/op 2014-06-29 19:26:34,345 INFO [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs$1(79): Received created event:/hbase/error injection coordination/abort/op 2014-06-29 19:26:34,345 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-------one 2014-06-29 19:26:34,345 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(304): Aborting procedure member for znode /hbase/error injection coordination/abort/op 2014-06-29 19:26:34,352 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-abort 2014-06-29 19:26:34,352 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |----op 2014-06-29 19:26:34,352 DEBUG [pool-1-thread-1-EventThread] procedure.ProcedureMember(241): Request received to abort procedure op org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via TIMER:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.errorhandling.ForeignException.deserialize(ForeignException.java:171) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.abort(ZKProcedureMemberRpcs.java:320) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs$1.nodeCreated(ZKProcedureMemberRpcs.java:94) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:327) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) Caused by: org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:244) at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:238) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34) at org.mockito.internal.MockHandler.handle(MockHandler.java:99) at org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36) at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.acquireBarrier() at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:161) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl.call(Subprocedure.java:315) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) 2014-06-29 19:26:34,353 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-reached 2014-06-29 19:26:34,353 ERROR [pool-1-thread-1-EventThread] procedure.ProcedureMember(250): Propagating foreign exception to subprocedure op org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via TIMER:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.errorhandling.ForeignException.deserialize(ForeignException.java:171) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.abort(ZKProcedureMemberRpcs.java:320) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs$1.nodeCreated(ZKProcedureMemberRpcs.java:94) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:327) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) Caused by: org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:244) at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:238) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34) at org.mockito.internal.MockHandler.handle(MockHandler.java:99) at org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36) at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.acquireBarrier() at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:161) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl.call(Subprocedure.java:315) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) 2014-06-29 19:26:34,353 DEBUG [pool-1-thread-1-EventThread] procedure.ProcedureMember(241): Request received to abort procedure op org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via TIMER:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.errorhandling.ForeignException.deserialize(ForeignException.java:171) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.abort(ZKProcedureMemberRpcs.java:320) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs$1.nodeCreated(ZKProcedureMemberRpcs.java:94) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:327) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) Caused by: org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:244) at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:238) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34) at org.mockito.internal.MockHandler.handle(MockHandler.java:99) at org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36) at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.acquireBarrier() at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:161) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl.call(Subprocedure.java:315) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) 2014-06-29 19:26:34,354 ERROR [pool-1-thread-1-EventThread] procedure.Subprocedure(265): Propagating foreign exception to subprocedure op org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via TIMER:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.errorhandling.ForeignException.deserialize(ForeignException.java:171) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.abort(ZKProcedureMemberRpcs.java:320) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs$1.nodeCreated(ZKProcedureMemberRpcs.java:94) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:327) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) Caused by: org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:244) at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:238) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34) at org.mockito.internal.MockHandler.handle(MockHandler.java:99) at org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36) at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.acquireBarrier() at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:161) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl.call(Subprocedure.java:315) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) 2014-06-29 19:26:34,355 DEBUG [pool-1-thread-1-EventThread] procedure.ProcedureMember(241): Request received to abort procedure op org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via TIMER:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.errorhandling.ForeignException.deserialize(ForeignException.java:171) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.abort(ZKProcedureMemberRpcs.java:320) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs$1.nodeCreated(ZKProcedureMemberRpcs.java:94) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:327) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) Caused by: org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:244) at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:238) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34) at org.mockito.internal.MockHandler.handle(MockHandler.java:99) at org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36) at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.acquireBarrier() at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:161) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl.call(Subprocedure.java:315) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) 2014-06-29 19:26:34,355 DEBUG [pool-1-thread-1-EventThread] errorhandling.ForeignExceptionDispatcher(66): accepting received exception org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via TIMER:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.errorhandling.ForeignException.deserialize(ForeignException.java:171) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.abort(ZKProcedureMemberRpcs.java:320) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs$1.nodeCreated(ZKProcedureMemberRpcs.java:94) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:327) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) Caused by: org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:244) at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:238) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34) at org.mockito.internal.MockHandler.handle(MockHandler.java:99) at org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36) at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.acquireBarrier() at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:161) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl.call(Subprocedure.java:315) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) 2014-06-29 19:26:34,354 ERROR [pool-1-thread-1-EventThread] procedure.ProcedureMember(250): Propagating foreign exception to subprocedure op org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via TIMER:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.errorhandling.ForeignException.deserialize(ForeignException.java:171) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.abort(ZKProcedureMemberRpcs.java:320) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs$1.nodeCreated(ZKProcedureMemberRpcs.java:94) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:327) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) Caused by: org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:244) at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:238) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34) at org.mockito.internal.MockHandler.handle(MockHandler.java:99) at org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36) at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.acquireBarrier() at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:161) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl.call(Subprocedure.java:315) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) 2014-06-29 19:26:34,354 DEBUG [pool-1-thread-1-EventThread] procedure.Procedure(294): member: 'one' joining acquired barrier for procedure 'op' on coordinator 2014-06-29 19:26:34,356 ERROR [pool-1-thread-1-EventThread] procedure.Subprocedure(265): Propagating foreign exception to subprocedure op org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via TIMER:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.errorhandling.ForeignException.deserialize(ForeignException.java:171) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.abort(ZKProcedureMemberRpcs.java:320) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs$1.nodeCreated(ZKProcedureMemberRpcs.java:94) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:327) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) Caused by: org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:244) at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:238) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34) at org.mockito.internal.MockHandler.handle(MockHandler.java:99) at org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36) at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.acquireBarrier() at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:161) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl.call(Subprocedure.java:315) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) 2014-06-29 19:26:34,356 DEBUG [pool-1-thread-1-EventThread] procedure.Subprocedure$1(106): Was remote foreign exception, not redispatching error org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via TIMER:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.errorhandling.ForeignException.deserialize(ForeignException.java:171) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.abort(ZKProcedureMemberRpcs.java:320) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs$1.nodeCreated(ZKProcedureMemberRpcs.java:94) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:327) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) Caused by: org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:244) at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:238) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34) at org.mockito.internal.MockHandler.handle(MockHandler.java:99) at org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36) at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.acquireBarrier() at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:161) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl.call(Subprocedure.java:315) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) 2014-06-29 19:26:34,355 ERROR [pool-1-thread-1-EventThread] procedure.ProcedureMember(250): Propagating foreign exception to subprocedure op org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via TIMER:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.errorhandling.ForeignException.deserialize(ForeignException.java:171) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.abort(ZKProcedureMemberRpcs.java:320) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs$1.nodeCreated(ZKProcedureMemberRpcs.java:94) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:327) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) Caused by: org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:244) at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:238) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34) at org.mockito.internal.MockHandler.handle(MockHandler.java:99) at org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36) at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.acquireBarrier() at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:161) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl.call(Subprocedure.java:315) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) 2014-06-29 19:26:34,357 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility-0x146e917cf2b0008, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/error injection coordination/abort 2014-06-29 19:26:34,358 ERROR [pool-1-thread-1-EventThread] procedure.Subprocedure(265): Propagating foreign exception to subprocedure op org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via TIMER:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.errorhandling.ForeignException.deserialize(ForeignException.java:171) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.abort(ZKProcedureMemberRpcs.java:320) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs$1.nodeCreated(ZKProcedureMemberRpcs.java:94) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:327) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) Caused by: org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:244) at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:238) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34) at org.mockito.internal.MockHandler.handle(MockHandler.java:99) at org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36) at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.acquireBarrier() at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:161) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl.call(Subprocedure.java:315) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) 2014-06-29 19:26:34,357 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility-0x146e917cf2b0009, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/error injection coordination/abort 2014-06-29 19:26:34,358 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility-0x146e917cf2b000a, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/error injection coordination/abort 2014-06-29 19:26:34,358 INFO [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs$1(109): Received procedure abort children changed event: /hbase/error injection coordination/abort 2014-06-29 19:26:34,356 DEBUG [pool-1-thread-1-EventThread] procedure.Procedure(303): Waiting on: java.util.concurrent.CountDownLatch@13dd208[Count = 2] remaining members to acquire global barrier 2014-06-29 19:26:34,359 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/error injection coordination/abort' 2014-06-29 19:26:34,359 INFO [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs$1(109): Received procedure abort children changed event: /hbase/error injection coordination/abort 2014-06-29 19:26:34,359 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/error injection coordination/abort' 2014-06-29 19:26:34,358 INFO [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs$1(109): Received procedure abort children changed event: /hbase/error injection coordination/abort 2014-06-29 19:26:34,359 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/error injection coordination/abort' 2014-06-29 19:26:34,359 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility-0x146e917cf2b0007, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/error injection coordination/abort/op 2014-06-29 19:26:34,360 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(170): Node created: /hbase/error injection coordination/abort/op 2014-06-29 19:26:34,360 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2014-06-29 19:26:34,360 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/error injection coordination 2014-06-29 19:26:34,360 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(304): Aborting procedure member for znode /hbase/error injection coordination/abort/op 2014-06-29 19:26:34,360 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(304): Aborting procedure member for znode /hbase/error injection coordination/abort/op 2014-06-29 19:26:34,361 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2014-06-29 19:26:34,361 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(304): Aborting procedure member for znode /hbase/error injection coordination/abort/op 2014-06-29 19:26:34,362 DEBUG [pool-1-thread-1-EventThread] procedure.ProcedureMember(241): Request received to abort procedure op org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via TIMER:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.errorhandling.ForeignException.deserialize(ForeignException.java:171) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.abort(ZKProcedureMemberRpcs.java:320) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.watchForAbortedProcedures(ZKProcedureMemberRpcs.java:142) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.access$200(ZKProcedureMemberRpcs.java:55) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs$1.nodeChildrenChanged(ZKProcedureMemberRpcs.java:110) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:348) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) Caused by: org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:244) at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:238) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34) at org.mockito.internal.MockHandler.handle(MockHandler.java:99) at org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36) at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.acquireBarrier() at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:161) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl.call(Subprocedure.java:315) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) 2014-06-29 19:26:34,362 DEBUG [pool-1-thread-1-EventThread] procedure.ProcedureMember(241): Request received to abort procedure op org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via TIMER:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.errorhandling.ForeignException.deserialize(ForeignException.java:171) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.abort(ZKProcedureMemberRpcs.java:320) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.watchForAbortedProcedures(ZKProcedureMemberRpcs.java:142) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.access$200(ZKProcedureMemberRpcs.java:55) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs$1.nodeChildrenChanged(ZKProcedureMemberRpcs.java:110) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:348) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) Caused by: org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:244) at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:238) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34) at org.mockito.internal.MockHandler.handle(MockHandler.java:99) at org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36) at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.acquireBarrier() at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:161) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl.call(Subprocedure.java:315) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) 2014-06-29 19:26:34,363 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |----op 2014-06-29 19:26:34,363 ERROR [pool-1-thread-1-EventThread] procedure.ProcedureMember(250): Propagating foreign exception to subprocedure op org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via TIMER:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.errorhandling.ForeignException.deserialize(ForeignException.java:171) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.abort(ZKProcedureMemberRpcs.java:320) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.watchForAbortedProcedures(ZKProcedureMemberRpcs.java:142) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.access$200(ZKProcedureMemberRpcs.java:55) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs$1.nodeChildrenChanged(ZKProcedureMemberRpcs.java:110) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:348) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) Caused by: org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:244) at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:238) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34) at org.mockito.internal.MockHandler.handle(MockHandler.java:99) at org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36) at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.acquireBarrier() at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:161) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl.call(Subprocedure.java:315) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) 2014-06-29 19:26:34,362 DEBUG [pool-1-thread-1-EventThread] procedure.ProcedureMember(241): Request received to abort procedure op org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via TIMER:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.errorhandling.ForeignException.deserialize(ForeignException.java:171) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.abort(ZKProcedureMemberRpcs.java:320) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.watchForAbortedProcedures(ZKProcedureMemberRpcs.java:142) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.access$200(ZKProcedureMemberRpcs.java:55) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs$1.nodeChildrenChanged(ZKProcedureMemberRpcs.java:110) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:348) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) Caused by: org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:244) at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:238) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34) at org.mockito.internal.MockHandler.handle(MockHandler.java:99) at org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36) at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.acquireBarrier() at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:161) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl.call(Subprocedure.java:315) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) 2014-06-29 19:26:34,362 ERROR [pool-1-thread-1-EventThread] procedure.ProcedureMember(250): Propagating foreign exception to subprocedure op org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via TIMER:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.errorhandling.ForeignException.deserialize(ForeignException.java:171) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.abort(ZKProcedureMemberRpcs.java:320) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.watchForAbortedProcedures(ZKProcedureMemberRpcs.java:142) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.access$200(ZKProcedureMemberRpcs.java:55) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs$1.nodeChildrenChanged(ZKProcedureMemberRpcs.java:110) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:348) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) Caused by: org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:244) at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:238) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34) at org.mockito.internal.MockHandler.handle(MockHandler.java:99) at org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36) at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.acquireBarrier() at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:161) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl.call(Subprocedure.java:315) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) 2014-06-29 19:26:34,364 ERROR [pool-1-thread-1-EventThread] procedure.ProcedureMember(250): Propagating foreign exception to subprocedure op org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via TIMER:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.errorhandling.ForeignException.deserialize(ForeignException.java:171) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.abort(ZKProcedureMemberRpcs.java:320) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.watchForAbortedProcedures(ZKProcedureMemberRpcs.java:142) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.access$200(ZKProcedureMemberRpcs.java:55) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs$1.nodeChildrenChanged(ZKProcedureMemberRpcs.java:110) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:348) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) Caused by: org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:244) at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:238) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34) at org.mockito.internal.MockHandler.handle(MockHandler.java:99) at org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36) at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.acquireBarrier() at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:161) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl.call(Subprocedure.java:315) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) 2014-06-29 19:26:34,363 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-------one 2014-06-29 19:26:34,363 ERROR [pool-1-thread-1-EventThread] procedure.Subprocedure(265): Propagating foreign exception to subprocedure op org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via TIMER:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.errorhandling.ForeignException.deserialize(ForeignException.java:171) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.abort(ZKProcedureMemberRpcs.java:320) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.watchForAbortedProcedures(ZKProcedureMemberRpcs.java:142) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.access$200(ZKProcedureMemberRpcs.java:55) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs$1.nodeChildrenChanged(ZKProcedureMemberRpcs.java:110) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:348) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) Caused by: org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:244) at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:238) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34) at org.mockito.internal.MockHandler.handle(MockHandler.java:99) at org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36) at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.acquireBarrier() at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:161) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl.call(Subprocedure.java:315) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) 2014-06-29 19:26:34,365 ERROR [pool-1-thread-1-EventThread] procedure.Subprocedure(265): Propagating foreign exception to subprocedure op org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via TIMER:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.errorhandling.ForeignException.deserialize(ForeignException.java:171) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.abort(ZKProcedureMemberRpcs.java:320) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.watchForAbortedProcedures(ZKProcedureMemberRpcs.java:142) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.access$200(ZKProcedureMemberRpcs.java:55) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs$1.nodeChildrenChanged(ZKProcedureMemberRpcs.java:110) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:348) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) Caused by: org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:244) at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:238) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34) at org.mockito.internal.MockHandler.handle(MockHandler.java:99) at org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36) at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.acquireBarrier() at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:161) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl.call(Subprocedure.java:315) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) 2014-06-29 19:26:34,364 ERROR [pool-1-thread-1-EventThread] procedure.Subprocedure(265): Propagating foreign exception to subprocedure op org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via TIMER:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.errorhandling.ForeignException.deserialize(ForeignException.java:171) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.abort(ZKProcedureMemberRpcs.java:320) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.watchForAbortedProcedures(ZKProcedureMemberRpcs.java:142) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.access$200(ZKProcedureMemberRpcs.java:55) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs$1.nodeChildrenChanged(ZKProcedureMemberRpcs.java:110) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:348) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) Caused by: org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:244) at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:238) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34) at org.mockito.internal.MockHandler.handle(MockHandler.java:99) at org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36) at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.acquireBarrier() at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:161) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl.call(Subprocedure.java:315) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) 2014-06-29 19:26:34,365 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-abort 2014-06-29 19:26:34,367 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |----op 2014-06-29 19:26:34,368 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-reached 2014-06-29 19:26:34,370 DEBUG [pool-1-thread-1-EventThread] errorhandling.ForeignExceptionDispatcher(66): accepting received exception org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via TIMER:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.errorhandling.ForeignException.deserialize(ForeignException.java:171) at org.apache.hadoop.hbase.procedure.ZKProcedureCoordinatorRpcs.abort(ZKProcedureCoordinatorRpcs.java:243) at org.apache.hadoop.hbase.procedure.ZKProcedureCoordinatorRpcs$1.nodeCreated(ZKProcedureCoordinatorRpcs.java:183) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:327) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) Caused by: org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:244) at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:238) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34) at org.mockito.internal.MockHandler.handle(MockHandler.java:99) at org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36) at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.acquireBarrier() at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:161) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl.call(Subprocedure.java:315) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) 2014-06-29 19:26:34,370 DEBUG [member: 'two' subprocedure-pool10-thread-1] procedure.Subprocedure(162): Subprocedure 'op' locally acquired 2014-06-29 19:26:34,371 DEBUG [member: 'two' subprocedure-pool10-thread-1] procedure.ZKProcedureMemberRpcs(237): Member: 'two' joining acquired barrier for procedure (op) in zk 2014-06-29 19:26:34,392 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility-0x146e917cf2b0007, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/error injection coordination/acquired/op/two 2014-06-29 19:26:34,392 DEBUG [member: 'two' subprocedure-pool10-thread-1] procedure.ZKProcedureMemberRpcs(245): Watch for global barrier reached:/hbase/error injection coordination/reached/op 2014-06-29 19:26:34,392 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(170): Node created: /hbase/error injection coordination/acquired/op/two 2014-06-29 19:26:34,393 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2014-06-29 19:26:34,393 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/error injection coordination 2014-06-29 19:26:34,393 DEBUG [member: 'two' subprocedure-pool10-thread-1] zookeeper.ZKUtil(430): testing utility-0x146e917cf2b0009, quorum=localhost:51439, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/error injection coordination/reached/op 2014-06-29 19:26:34,393 DEBUG [member: 'two' subprocedure-pool10-thread-1] procedure.Subprocedure(166): Subprocedure 'op' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2014-06-29 19:26:34,393 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2014-06-29 19:26:34,394 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |----op 2014-06-29 19:26:34,395 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-------two 2014-06-29 19:26:34,395 ERROR [member: 'two' subprocedure-pool10-thread-1] procedure.Subprocedure(265): Subprocedure 'op' aborting due to a ForeignException! org.apache.hadoop.hbase.errorhandling.TimeoutException via TIMER:org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.errorhandling.ForeignExceptionDispatcher.rethrowException(ForeignExceptionDispatcher.java:83) at org.apache.hadoop.hbase.procedure.Procedure.waitForLatch(Procedure.java:369) at org.apache.hadoop.hbase.procedure.Subprocedure.waitForReachedGlobalBarrier(Subprocedure.java:296) at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:170) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl.call(Subprocedure.java:315) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) Caused by: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:244) at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:238) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34) at org.mockito.internal.MockHandler.handle(MockHandler.java:99) at org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36) at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.acquireBarrier() at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:161) ... 6 more 2014-06-29 19:26:34,395 DEBUG [member: 'two' subprocedure-pool10-thread-1] procedure.Subprocedure(203): Subprocedure 'op' running cleanup. 2014-06-29 19:26:34,395 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-------one 2014-06-29 19:26:34,396 DEBUG [member: 'two' subprocedure-pool10-thread-1] errorhandling.TimeoutExceptionInjector(89): Marking timer as complete - no error notifications will be received for this timer. 2014-06-29 19:26:34,396 DEBUG [member: 'two' subprocedure-pool10-thread-1] procedure.Subprocedure(211): Subprocedure 'op' completed. 2014-06-29 19:26:34,396 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-abort 2014-06-29 19:26:34,397 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |----op 2014-06-29 19:26:34,397 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureUtil(263): |-reached 2014-06-29 19:26:34,398 DEBUG [pool-1-thread-1-EventThread] procedure.Procedure(294): member: 'two' joining acquired barrier for procedure 'op' on coordinator 2014-06-29 19:26:34,398 DEBUG [pool-1-thread-1-EventThread] procedure.Procedure(303): Waiting on: java.util.concurrent.CountDownLatch@13dd208[Count = 1] remaining members to acquire global barrier 2014-06-29 19:26:34,579 DEBUG [pool-1-thread-1] procedure.TestZKProcedure(392): Checking mock:0 2014-06-29 19:26:34,598 ERROR [(coordinator)-proc-coordinator-pool8-thread-1] procedure.Procedure(223): Procedure 'op' execution failed! org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via TIMER:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.errorhandling.ForeignExceptionDispatcher.rethrowException(ForeignExceptionDispatcher.java:83) at org.apache.hadoop.hbase.procedure.Procedure.waitForLatch(Procedure.java:369) at org.apache.hadoop.hbase.procedure.Procedure.call(Procedure.java:206) at org.apache.hadoop.hbase.procedure.Procedure.call(Procedure.java:67) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) Caused by: org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:244) at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:238) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34) at org.mockito.internal.MockHandler.handle(MockHandler.java:99) at org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36) at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.acquireBarrier() at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:161) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl.call(Subprocedure.java:315) ... 5 more 2014-06-29 19:26:34,598 DEBUG [(coordinator)-proc-coordinator-pool8-thread-1] procedure.Procedure(226): Running finish phase. 2014-06-29 19:26:34,599 DEBUG [(coordinator)-proc-coordinator-pool8-thread-1] procedure.Procedure(277): Finished coordinator procedure - removing self from list of running procedures 2014-06-29 19:26:34,599 DEBUG [(coordinator)-proc-coordinator-pool8-thread-1] procedure.ZKProcedureCoordinatorRpcs(141): Attempting to clean out zk node for op:op 2014-06-29 19:26:34,599 INFO [(coordinator)-proc-coordinator-pool8-thread-1] procedure.ZKProcedureUtil(284): Clearing all znodes for procedure opincluding nodes /hbase/error injection coordination/acquired /hbase/error injection coordination/reached /hbase/error injection coordination/abort 2014-06-29 19:26:34,603 INFO [pool-1-thread-1] hbase.ResourceChecker(171): after: procedure.TestZKProcedure#testMultiCohortWithMemberTimeoutDuringPrepare Thread=38 (was 24) Potentially hanging thread: Timer-9 java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:485) java.util.TimerThread.mainLoop(Timer.java:483) java.util.TimerThread.run(Timer.java:462) Potentially hanging thread: member: 'two' subprocedure-pool10-thread-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: member: 'three' subprocedure-pool11-thread-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1011) java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1303) java.util.concurrent.CountDownLatch.await(CountDownLatch.java:253) org.apache.hadoop.hbase.procedure.Procedure.waitForLatch(Procedure.java:374) org.apache.hadoop.hbase.procedure.Subprocedure.waitForReachedGlobalBarrier(Subprocedure.java:296) org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.CGLIB$waitForReachedGlobalBarrier$11() org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547$$FastClassByMockitoWithCGLIB$$cd77604e.invoke() org.mockito.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:216) org.mockito.internal.creation.AbstractMockitoMethodProxy.invokeSuper(AbstractMockitoMethodProxy.java:10) org.mockito.internal.invocation.realmethod.CGLIBProxyRealMethod.invoke(CGLIBProxyRealMethod.java:22) org.mockito.internal.invocation.realmethod.FilteredCGLIBProxyRealMethod.invoke(FilteredCGLIBProxyRealMethod.java:27) org.mockito.internal.invocation.Invocation.callRealMethod(Invocation.java:213) org.mockito.internal.stubbing.answers.CallsRealMethods.answer(CallsRealMethods.java:36) org.mockito.internal.MockHandler.handle(MockHandler.java:101) org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36) org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48) org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.waitForReachedGlobalBarrier() org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:170) org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl.call(Subprocedure.java:315) java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) java.util.concurrent.FutureTask.run(FutureTask.java:138) java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: member: 'one' subprocedure-pool9-thread-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1011) java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1303) java.util.concurrent.CountDownLatch.await(CountDownLatch.java:253) org.apache.hadoop.hbase.procedure.Procedure.waitForLatch(Procedure.java:374) org.apache.hadoop.hbase.procedure.Subprocedure.waitForReachedGlobalBarrier(Subprocedure.java:296) org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.CGLIB$waitForReachedGlobalBarrier$11() org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547$$FastClassByMockitoWithCGLIB$$cd77604e.invoke() org.mockito.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:216) org.mockito.internal.creation.AbstractMockitoMethodProxy.invokeSuper(AbstractMockitoMethodProxy.java:10) org.mockito.internal.invocation.realmethod.CGLIBProxyRealMethod.invoke(CGLIBProxyRealMethod.java:22) org.mockito.internal.invocation.realmethod.FilteredCGLIBProxyRealMethod.invoke(FilteredCGLIBProxyRealMethod.java:27) org.mockito.internal.invocation.Invocation.callRealMethod(Invocation.java:213) org.mockito.internal.stubbing.answers.CallsRealMethods.answer(CallsRealMethods.java:36) org.mockito.internal.MockHandler.handle(MockHandler.java:101) org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36) org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48) org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.waitForReachedGlobalBarrier() org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:170) org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl.call(Subprocedure.java:315) java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) java.util.concurrent.FutureTask.run(FutureTask.java:138) java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: (coordinator)-proc-coordinator-pool8-thread-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1011) java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1303) java.util.concurrent.CountDownLatch.await(CountDownLatch.java:253) org.apache.hadoop.hbase.procedure.Procedure.waitForLatch(Procedure.java:374) org.apache.hadoop.hbase.procedure.Procedure.call(Procedure.java:206) org.apache.hadoop.hbase.procedure.Procedure.call(Procedure.java:67) java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) java.util.concurrent.FutureTask.run(FutureTask.java:138) java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: Timer-7 java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:509) java.util.TimerThread.run(Timer.java:462) Potentially hanging thread: Timer-10 java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:509) java.util.TimerThread.run(Timer.java:462) - Thread LEAK? -, OpenFileDescriptor=187 (was 173) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=40000 (was 40000), SystemLoadAverage=360 (was 360), ProcessCount=168 (was 168), AvailableMemoryMB=3166 (was 3168), ConnectionCount=0 (was 0) 2014-06-29 19:26:34,631 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility-0x146e917cf2b0008, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/error injection coordination/acquired 2014-06-29 19:26:34,632 INFO [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs$1(106): Received procedure start children changed event: /hbase/error injection coordination/acquired 2014-06-29 19:26:34,632 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility-0x146e917cf2b0009, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/error injection coordination/acquired 2014-06-29 19:26:34,632 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility-0x146e917cf2b000a, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/error injection coordination/acquired 2014-06-29 19:26:34,632 INFO [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs$1(106): Received procedure start children changed event: /hbase/error injection coordination/acquired 2014-06-29 19:26:34,632 INFO [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs$1(106): Received procedure start children changed event: /hbase/error injection coordination/acquired 2014-06-29 19:26:34,632 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/error injection coordination/acquired' 2014-06-29 19:26:34,633 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/error injection coordination/acquired' 2014-06-29 19:26:34,632 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/error injection coordination/acquired' 2014-06-29 19:26:34,638 ERROR [member: 'three' subprocedure-pool11-thread-1] procedure.Subprocedure(265): Subprocedure 'op' aborting due to a ForeignException! org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via TIMER:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.errorhandling.ForeignExceptionDispatcher.rethrowException(ForeignExceptionDispatcher.java:83) at org.apache.hadoop.hbase.procedure.Procedure.waitForLatch(Procedure.java:369) at org.apache.hadoop.hbase.procedure.Subprocedure.waitForReachedGlobalBarrier(Subprocedure.java:296) at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:170) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl.call(Subprocedure.java:315) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) Caused by: org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:244) at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:238) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34) at org.mockito.internal.MockHandler.handle(MockHandler.java:99) at org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36) at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.acquireBarrier() at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:161) ... 6 more 2014-06-29 19:26:34,638 DEBUG [member: 'three' subprocedure-pool11-thread-1] procedure.Subprocedure(203): Subprocedure 'op' running cleanup. 2014-06-29 19:26:34,638 DEBUG [member: 'three' subprocedure-pool11-thread-1] errorhandling.TimeoutExceptionInjector(89): Marking timer as complete - no error notifications will be received for this timer. 2014-06-29 19:26:34,638 DEBUG [member: 'three' subprocedure-pool11-thread-1] procedure.Subprocedure(211): Subprocedure 'op' completed. 2014-06-29 19:26:34,639 ERROR [member: 'one' subprocedure-pool9-thread-1] procedure.Subprocedure(265): Subprocedure 'op' aborting due to a ForeignException! org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via TIMER:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.errorhandling.ForeignExceptionDispatcher.rethrowException(ForeignExceptionDispatcher.java:83) at org.apache.hadoop.hbase.procedure.Procedure.waitForLatch(Procedure.java:369) at org.apache.hadoop.hbase.procedure.Subprocedure.waitForReachedGlobalBarrier(Subprocedure.java:296) at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:170) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl.call(Subprocedure.java:315) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) Caused by: org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:subprocTimeout Start:1, End:2, diff:1, max:0 ms at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:244) at org.apache.hadoop.hbase.procedure.TestZKProcedure$4.answer(TestZKProcedure.java:238) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34) at org.mockito.internal.MockHandler.handle(MockHandler.java:99) at org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36) at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48) at org.apache.hadoop.hbase.procedure.Subprocedure$SubprocedureImpl$$EnhancerByMockitoWithCGLIB$$d2a0d547.acquireBarrier() at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:161) ... 6 more 2014-06-29 19:26:34,639 DEBUG [member: 'one' subprocedure-pool9-thread-1] procedure.Subprocedure(203): Subprocedure 'op' running cleanup. 2014-06-29 19:26:34,639 WARN [member: 'one' subprocedure-pool9-thread-1] errorhandling.TimeoutExceptionInjector(86): Timer already marked completed, ignoring! 2014-06-29 19:26:34,639 DEBUG [member: 'one' subprocedure-pool9-thread-1] procedure.Subprocedure(211): Subprocedure 'op' completed. 2014-06-29 19:26:34,664 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility-0x146e917cf2b000a, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/error injection coordination/abort 2014-06-29 19:26:34,665 DEBUG [(coordinator)-proc-coordinator-pool8-thread-1] errorhandling.TimeoutExceptionInjector(89): Marking timer as complete - no error notifications will be received for this timer. 2014-06-29 19:26:34,665 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility-0x146e917cf2b0008, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/error injection coordination/abort 2014-06-29 19:26:34,665 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): testing utility-0x146e917cf2b0009, quorum=localhost:51439, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/error injection coordination/abort 2014-06-29 19:26:34,665 INFO [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs$1(109): Received procedure abort children changed event: /hbase/error injection coordination/abort 2014-06-29 19:26:34,666 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/error injection coordination/abort' 2014-06-29 19:26:34,665 INFO [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs$1(109): Received procedure abort children changed event: /hbase/error injection coordination/abort 2014-06-29 19:26:34,665 INFO [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs$1(109): Received procedure abort children changed event: /hbase/error injection coordination/abort 2014-06-29 19:26:34,666 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/error injection coordination/abort' 2014-06-29 19:26:34,666 DEBUG [pool-1-thread-1-EventThread] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/error injection coordination/abort'