2012-12-28 18:00:05,782 INFO [main] hbase.ResourceChecker(157): before regionserver.TestSplitLogWorker#testAcquireTaskAtStartup: 4 threads, 81 file descriptors 0 connections, 2012-12-28 18:00:05,968 INFO [main] zookeeper.MiniZooKeeperCluster(193): Started MiniZK Cluster and connect 1 ZK server on client port: 65127 2012-12-28 18:00:05,986 DEBUG [main] zookeeper.ZKUtil(120): split-log-worker-tests opening connection to ZooKeeper with ensemble (localhost:65127) 2012-12-28 18:00:06,005 INFO [main] zookeeper.RecoverableZooKeeper(104): The identifier of this process is 15724@acer.localdomain 2012-12-28 18:00:06,118 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(277): split-log-worker-tests Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2012-12-28 18:00:06,125 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(354): split-log-worker-tests-0x13be463380c0000 connected 2012-12-28 18:00:06,242 DEBUG [main] regionserver.TestSplitLogWorker(97): /hbase created 2012-12-28 18:00:06,342 DEBUG [main] regionserver.TestSplitLogWorker(100): /hbase/splitlog created 2012-12-28 18:00:06,344 INFO [main] regionserver.TestSplitLogWorker(131): testAcquireTaskAtStartup 2012-12-28 18:00:06,445 INFO [SplitLogWorker-rs] regionserver.SplitLogWorker(138): SplitLogWorker rs starting 2012-12-28 18:00:06,464 DEBUG [SplitLogWorker-rs] zookeeper.ZKUtil(1541): split-log-worker-tests-0x13be463380c0000 Retrieved 14 byte(s) of data from znode /hbase/splitlog/tatas; data=unassigned mgr 2012-12-28 18:00:06,546 INFO [main] regionserver.SplitLogWorker(506): Sending interrupt to stop the worker thread 2012-12-28 18:00:06,547 WARN [SplitLogWorker-rs] regionserver.SplitLogWorker(352): Interrupted while trying to assert ownership of /hbase/splitlog/tatas java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1309) at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1264) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.setData(RecoverableZooKeeper.java:349) at org.apache.hadoop.hbase.regionserver.SplitLogWorker.attemptToOwnTask(SplitLogWorker.java:330) at org.apache.hadoop.hbase.regionserver.SplitLogWorker.grabTask(SplitLogWorker.java:248) at org.apache.hadoop.hbase.regionserver.SplitLogWorker.taskLoop(SplitLogWorker.java:195) at org.apache.hadoop.hbase.regionserver.SplitLogWorker.run(SplitLogWorker.java:163) at java.lang.Thread.run(Thread.java:679) 2012-12-28 18:00:06,547 INFO [SplitLogWorker-rs] regionserver.SplitLogWorker(169): SplitLogWorker rs exiting 2012-12-28 18:00:06,586 ERROR [SyncThread:0] server.NIOServerCnxn(180): Unexpected Exception: java.nio.channels.CancelledKeyException at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73) at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77) at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:153) at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1076) at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:404) at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:167) at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:101) 2012-12-28 18:00:06,589 INFO [main] zookeeper.MiniZooKeeperCluster(235): Shutdown MiniZK cluster with all ZK servers 2012-12-28 18:00:06,595 INFO [main] hbase.ResourceChecker(157): after regionserver.TestSplitLogWorker#testAcquireTaskAtStartup: 7 threads (was 4), 89 file descriptors (was 81). 0 connections, -thread leak?- -file handle leak?- 2012-12-28 18:00:06,596 INFO [main] hbase.ResourceChecker(182): after regionserver.TestSplitLogWorker#testAcquireTaskAtStartup: potentially hanging thread - SessionTracker 2012-12-28 18:00:06,597 INFO [main] hbase.ResourceChecker(185): java.lang.Object.wait(Native Method) 2012-12-28 18:00:06,597 INFO [main] hbase.ResourceChecker(185): org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:146) 2012-12-28 18:00:06,597 INFO [main] hbase.ResourceChecker(182): after regionserver.TestSplitLogWorker#testAcquireTaskAtStartup: potentially hanging thread - main-EventThread 2012-12-28 18:00:06,598 INFO [main] hbase.ResourceChecker(185): sun.misc.Unsafe.park(Native Method) 2012-12-28 18:00:06,598 INFO [main] hbase.ResourceChecker(185): java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) 2012-12-28 18:00:06,599 INFO [main] hbase.ResourceChecker(185): java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) 2012-12-28 18:00:06,599 INFO [main] hbase.ResourceChecker(185): java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:386) 2012-12-28 18:00:06,600 INFO [main] hbase.ResourceChecker(185): org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491) 2012-12-28 18:00:06,600 INFO [main] hbase.ResourceChecker(182): after regionserver.TestSplitLogWorker#testAcquireTaskAtStartup: potentially hanging thread - main-SendThread(localhost.localdomain:65127) 2012-12-28 18:00:06,600 INFO [main] hbase.ResourceChecker(185): java.lang.Thread.sleep(Native Method) 2012-12-28 18:00:06,601 INFO [main] hbase.ResourceChecker(185): org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:219) 2012-12-28 18:00:06,601 INFO [main] hbase.ResourceChecker(185): org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1157) 2012-12-28 18:00:06,602 INFO [main] hbase.ResourceChecker(185): org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1097) 2012-12-28 18:00:06,609 INFO [main] hbase.ResourceChecker(157): before regionserver.TestSplitLogWorker#testRaceForTask: 7 threads, 89 file descriptors 0 connections, 2012-12-28 18:00:06,620 INFO [main] zookeeper.MiniZooKeeperCluster(193): Started MiniZK Cluster and connect 1 ZK server on client port: 64941 2012-12-28 18:00:06,621 DEBUG [main] zookeeper.ZKUtil(120): split-log-worker-tests opening connection to ZooKeeper with ensemble (localhost:64941) 2012-12-28 18:00:06,622 INFO [main] zookeeper.RecoverableZooKeeper(104): The identifier of this process is 15724@acer.localdomain 2012-12-28 18:00:06,651 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(277): split-log-worker-tests-0x13be463380c0000 Received ZooKeeper Event, type=None, state=Disconnected, path=null 2012-12-28 18:00:06,651 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(375): split-log-worker-tests-0x13be463380c0000 Received Disconnected from ZooKeeper, ignoring 2012-12-28 18:00:06,698 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(277): split-log-worker-tests Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2012-12-28 18:00:06,700 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(354): split-log-worker-tests-0x13be4633ada0000 connected 2012-12-28 18:00:06,755 DEBUG [main] regionserver.TestSplitLogWorker(97): /hbase created 2012-12-28 18:00:06,811 DEBUG [main] regionserver.TestSplitLogWorker(100): /hbase/splitlog created 2012-12-28 18:00:06,812 INFO [main] regionserver.TestSplitLogWorker(163): testRaceForTask 2012-12-28 18:00:06,866 INFO [SplitLogWorker-svr1] regionserver.SplitLogWorker(138): SplitLogWorker svr1 starting 2012-12-28 18:00:06,866 INFO [SplitLogWorker-svr2] regionserver.SplitLogWorker(138): SplitLogWorker svr2 starting 2012-12-28 18:00:06,872 DEBUG [SplitLogWorker-svr1] zookeeper.ZKUtil(1541): split-log-worker-tests-0x13be4633ada0000 Retrieved 18 byte(s) of data from znode /hbase/splitlog/trft; data=unassigned manager 2012-12-28 18:00:06,873 DEBUG [SplitLogWorker-svr2] zookeeper.ZKUtil(1541): split-log-worker-tests-0x13be4633ada0000 Retrieved 18 byte(s) of data from znode /hbase/splitlog/trft; data=unassigned manager 2012-12-28 18:00:06,933 INFO [SplitLogWorker-svr1] regionserver.SplitLogWorker(257): worker svr1 acquired task /hbase/splitlog/trft 2012-12-28 18:00:08,001 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(277): split-log-worker-tests-0x13be4633ada0000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitlog/trft 2012-12-28 18:00:08,002 DEBUG [main] zookeeper.ZKUtil(1541): split-log-worker-tests-0x13be4633ada0000 Retrieved 10 byte(s) of data from znode /hbase/splitlog/trft; data=owned svr1 2012-12-28 18:00:08,003 INFO [main] regionserver.SplitLogWorker(506): Sending interrupt to stop the worker thread 2012-12-28 18:00:08,005 WARN [SplitLogWorker-svr1] regionserver.SplitLogWorker(282): task execution prempted /hbase/splitlog/trft 2012-12-28 18:00:08,005 INFO [SplitLogWorker-svr1] regionserver.SplitLogWorker(306): worker svr1 done with task /hbase/splitlog/trft in 1071ms 2012-12-28 18:00:08,005 INFO [SplitLogWorker-svr1] regionserver.SplitLogWorker(169): SplitLogWorker svr1 exiting 2012-12-28 18:00:08,006 INFO [main] regionserver.SplitLogWorker(506): Sending interrupt to stop the worker thread 2012-12-28 18:00:08,006 INFO [SplitLogWorker-svr2] regionserver.SplitLogWorker(205): SplitLogWorker interrupted while waiting for task, exiting: java.lang.InterruptedException 2012-12-28 18:00:08,007 INFO [SplitLogWorker-svr2] regionserver.SplitLogWorker(169): SplitLogWorker svr2 exiting 2012-12-28 18:00:08,009 INFO [main] zookeeper.MiniZooKeeperCluster(235): Shutdown MiniZK cluster with all ZK servers 2012-12-28 18:00:08,011 INFO [main] hbase.ResourceChecker(157): after regionserver.TestSplitLogWorker#testRaceForTask: 9 threads (was 7), 94 file descriptors (was 89). 0 connections, -thread leak?- -file handle leak?- 2012-12-28 18:00:08,012 INFO [main] hbase.ResourceChecker(182): after regionserver.TestSplitLogWorker#testRaceForTask: potentially hanging thread - main-SendThread(localhost.localdomain:64941) 2012-12-28 18:00:08,012 INFO [main] hbase.ResourceChecker(185): java.lang.Thread.sleep(Native Method) 2012-12-28 18:00:08,013 INFO [main] hbase.ResourceChecker(185): org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:219) 2012-12-28 18:00:08,014 INFO [main] hbase.ResourceChecker(185): org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1157) 2012-12-28 18:00:08,018 INFO [main] hbase.ResourceChecker(185): org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1097) 2012-12-28 18:00:08,025 INFO [main] hbase.ResourceChecker(157): before regionserver.TestSplitLogWorker#testPreemptTask: 9 threads, 94 file descriptors 0 connections, 2012-12-28 18:00:08,032 INFO [main] zookeeper.MiniZooKeeperCluster(193): Started MiniZK Cluster and connect 1 ZK server on client port: 60685 2012-12-28 18:00:08,034 DEBUG [main] zookeeper.ZKUtil(120): split-log-worker-tests opening connection to ZooKeeper with ensemble (localhost:60685) 2012-12-28 18:00:08,040 INFO [main] zookeeper.RecoverableZooKeeper(104): The identifier of this process is 15724@acer.localdomain 2012-12-28 18:00:08,104 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(277): split-log-worker-tests Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2012-12-28 18:00:08,105 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(354): split-log-worker-tests-0x13be463405e0000 connected 2012-12-28 18:00:08,108 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(277): split-log-worker-tests-0x13be4633ada0000 Received ZooKeeper Event, type=None, state=Disconnected, path=null 2012-12-28 18:00:08,109 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(375): split-log-worker-tests-0x13be4633ada0000 Received Disconnected from ZooKeeper, ignoring 2012-12-28 18:00:08,158 DEBUG [main] regionserver.TestSplitLogWorker(97): /hbase created 2012-12-28 18:00:08,214 DEBUG [main] regionserver.TestSplitLogWorker(100): /hbase/splitlog created 2012-12-28 18:00:08,214 INFO [main] regionserver.TestSplitLogWorker(194): testPreemptTask 2012-12-28 18:00:08,215 INFO [SplitLogWorker-tpt_svr] regionserver.SplitLogWorker(138): SplitLogWorker tpt_svr starting 2012-12-28 18:00:08,379 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(277): split-log-worker-tests-0x13be463405e0000 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitlog 2012-12-28 18:00:08,379 DEBUG [main-EventThread] regionserver.SplitLogWorker(493): tasks arrived or departed 2012-12-28 18:00:08,382 DEBUG [SplitLogWorker-tpt_svr] zookeeper.ZKUtil(1541): split-log-worker-tests-0x13be463405e0000 Retrieved 18 byte(s) of data from znode /hbase/splitlog/tpt_task; data=unassigned manager 2012-12-28 18:00:08,435 INFO [SplitLogWorker-tpt_svr] regionserver.SplitLogWorker(257): worker tpt_svr acquired task /hbase/splitlog/tpt_task 2012-12-28 18:00:08,441 DEBUG [main] zookeeper.ZKUtil(1541): split-log-worker-tests-0x13be463405e0000 Retrieved 13 byte(s) of data from znode /hbase/splitlog/tpt_task; data=owned tpt_svr 2012-12-28 18:00:08,502 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(277): split-log-worker-tests-0x13be463405e0000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitlog/tpt_task 2012-12-28 18:00:08,503 INFO [main-EventThread] regionserver.SplitLogWorker(415): task /hbase/splitlog/tpt_task preempted from tpt_svr, current task state and owner=unassigned manager 2012-12-28 18:00:08,504 INFO [main-EventThread] regionserver.SplitLogWorker(506): Sending interrupt to stop the worker thread 2012-12-28 18:00:08,504 WARN [SplitLogWorker-tpt_svr] regionserver.SplitLogWorker(282): task execution prempted /hbase/splitlog/tpt_task 2012-12-28 18:00:08,504 INFO [SplitLogWorker-tpt_svr] regionserver.SplitLogWorker(306): worker tpt_svr done with task /hbase/splitlog/tpt_task in 69ms 2012-12-28 18:00:08,512 INFO [main] regionserver.SplitLogWorker(506): Sending interrupt to stop the worker thread 2012-12-28 18:00:08,513 INFO [SplitLogWorker-tpt_svr] regionserver.SplitLogWorker(205): SplitLogWorker interrupted while waiting for task, exiting: java.lang.InterruptedException 2012-12-28 18:00:08,513 INFO [SplitLogWorker-tpt_svr] regionserver.SplitLogWorker(169): SplitLogWorker tpt_svr exiting 2012-12-28 18:00:08,515 INFO [main] zookeeper.MiniZooKeeperCluster(235): Shutdown MiniZK cluster with all ZK servers 2012-12-28 18:00:08,517 INFO [main] hbase.ResourceChecker(157): after regionserver.TestSplitLogWorker#testPreemptTask: 12 threads (was 9), 98 file descriptors (was 94). 0 connections, -thread leak?- -file handle leak?- 2012-12-28 18:00:08,517 INFO [main] hbase.ResourceChecker(182): after regionserver.TestSplitLogWorker#testPreemptTask: potentially hanging thread - main-SendThread(localhost.localdomain:60685) 2012-12-28 18:00:08,518 INFO [main] hbase.ResourceChecker(185): java.lang.Thread.sleep(Native Method) 2012-12-28 18:00:08,518 INFO [main] hbase.ResourceChecker(185): org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:219) 2012-12-28 18:00:08,518 INFO [main] hbase.ResourceChecker(185): org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1157) 2012-12-28 18:00:08,519 INFO [main] hbase.ResourceChecker(185): org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1097) 2012-12-28 18:00:08,522 INFO [main] hbase.ResourceChecker(157): before regionserver.TestSplitLogWorker#testMultipleTasks: 12 threads, 98 file descriptors 0 connections, 2012-12-28 18:00:08,529 INFO [main] zookeeper.MiniZooKeeperCluster(193): Started MiniZK Cluster and connect 1 ZK server on client port: 61662 2012-12-28 18:00:08,531 DEBUG [main] zookeeper.ZKUtil(120): split-log-worker-tests opening connection to ZooKeeper with ensemble (localhost:61662) 2012-12-28 18:00:08,533 INFO [main] zookeeper.RecoverableZooKeeper(104): The identifier of this process is 15724@acer.localdomain 2012-12-28 18:00:08,581 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(277): split-log-worker-tests Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2012-12-28 18:00:08,582 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(354): split-log-worker-tests-0x13be463424e0000 connected 2012-12-28 18:00:08,614 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(277): split-log-worker-tests-0x13be463405e0000 Received ZooKeeper Event, type=None, state=Disconnected, path=null 2012-12-28 18:00:08,614 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(375): split-log-worker-tests-0x13be463405e0000 Received Disconnected from ZooKeeper, ignoring 2012-12-28 18:00:08,626 DEBUG [main] regionserver.TestSplitLogWorker(97): /hbase created 2012-12-28 18:00:08,671 DEBUG [main] regionserver.TestSplitLogWorker(100): /hbase/splitlog created 2012-12-28 18:00:08,671 INFO [main] regionserver.TestSplitLogWorker(224): testMultipleTasks 2012-12-28 18:00:08,672 INFO [SplitLogWorker-tmt_svr] regionserver.SplitLogWorker(138): SplitLogWorker tmt_svr starting 2012-12-28 18:00:08,816 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(277): split-log-worker-tests-0x13be463424e0000 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitlog 2012-12-28 18:00:08,816 DEBUG [main-EventThread] regionserver.SplitLogWorker(493): tasks arrived or departed 2012-12-28 18:00:08,819 DEBUG [SplitLogWorker-tmt_svr] zookeeper.ZKUtil(1541): split-log-worker-tests-0x13be463424e0000 Retrieved 18 byte(s) of data from znode /hbase/splitlog/tmt_task; data=unassigned manager 2012-12-28 18:00:08,860 INFO [SplitLogWorker-tmt_svr] regionserver.SplitLogWorker(257): worker tmt_svr acquired task /hbase/splitlog/tmt_task 2012-12-28 18:00:08,905 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(277): split-log-worker-tests-0x13be463424e0000 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitlog 2012-12-28 18:00:08,905 DEBUG [main-EventThread] regionserver.SplitLogWorker(493): tasks arrived or departed 2012-12-28 18:00:08,949 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(277): split-log-worker-tests-0x13be463424e0000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitlog/tmt_task 2012-12-28 18:00:08,951 INFO [main-EventThread] regionserver.SplitLogWorker(415): task /hbase/splitlog/tmt_task preempted from tmt_svr, current task state and owner=owned another-worker 2012-12-28 18:00:08,951 INFO [main-EventThread] regionserver.SplitLogWorker(506): Sending interrupt to stop the worker thread 2012-12-28 18:00:08,951 WARN [SplitLogWorker-tmt_svr] regionserver.SplitLogWorker(282): task execution prempted /hbase/splitlog/tmt_task 2012-12-28 18:00:08,952 INFO [SplitLogWorker-tmt_svr] regionserver.SplitLogWorker(306): worker tmt_svr done with task /hbase/splitlog/tmt_task in 91ms 2012-12-28 18:00:08,954 DEBUG [SplitLogWorker-tmt_svr] zookeeper.ZKUtil(1541): split-log-worker-tests-0x13be463424e0000 Retrieved 20 byte(s) of data from znode /hbase/splitlog/tmt_task; data=owned another-worker 2012-12-28 18:00:08,956 DEBUG [SplitLogWorker-tmt_svr] zookeeper.ZKUtil(1541): split-log-worker-tests-0x13be463424e0000 Retrieved 18 byte(s) of data from znode /hbase/splitlog/tmt_task_2; data=unassigned manager 2012-12-28 18:00:08,994 INFO [SplitLogWorker-tmt_svr] regionserver.SplitLogWorker(257): worker tmt_svr acquired task /hbase/splitlog/tmt_task_2 2012-12-28 18:00:09,002 DEBUG [main] zookeeper.ZKUtil(1541): split-log-worker-tests-0x13be463424e0000 Retrieved 13 byte(s) of data from znode /hbase/splitlog/tmt_task_2; data=owned tmt_svr 2012-12-28 18:00:09,002 INFO [main] regionserver.SplitLogWorker(506): Sending interrupt to stop the worker thread 2012-12-28 18:00:09,002 WARN [SplitLogWorker-tmt_svr] regionserver.SplitLogWorker(282): task execution prempted /hbase/splitlog/tmt_task_2 2012-12-28 18:00:09,003 INFO [SplitLogWorker-tmt_svr] regionserver.SplitLogWorker(306): worker tmt_svr done with task /hbase/splitlog/tmt_task_2 in 9ms 2012-12-28 18:00:09,003 INFO [SplitLogWorker-tmt_svr] regionserver.SplitLogWorker(169): SplitLogWorker tmt_svr exiting 2012-12-28 18:00:09,005 INFO [main] zookeeper.MiniZooKeeperCluster(235): Shutdown MiniZK cluster with all ZK servers 2012-12-28 18:00:09,007 INFO [main] hbase.ResourceChecker(157): after regionserver.TestSplitLogWorker#testMultipleTasks: 15 threads (was 12), 103 file descriptors (was 98). 0 connections, -thread leak?- -file handle leak?- 2012-12-28 18:00:09,007 INFO [main] hbase.ResourceChecker(182): after regionserver.TestSplitLogWorker#testMultipleTasks: potentially hanging thread - main-SendThread(localhost.localdomain:61662) 2012-12-28 18:00:09,008 INFO [main] hbase.ResourceChecker(185): java.lang.Thread.sleep(Native Method) 2012-12-28 18:00:09,008 INFO [main] hbase.ResourceChecker(185): org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:219) 2012-12-28 18:00:09,008 INFO [main] hbase.ResourceChecker(185): org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1157) 2012-12-28 18:00:09,009 INFO [main] hbase.ResourceChecker(185): org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1097) 2012-12-28 18:00:09,012 INFO [main] hbase.ResourceChecker(157): before regionserver.TestSplitLogWorker#testRescan: 15 threads, 103 file descriptors 0 connections, 2012-12-28 18:00:09,019 INFO [main] zookeeper.MiniZooKeeperCluster(193): Started MiniZK Cluster and connect 1 ZK server on client port: 52668 2012-12-28 18:00:09,021 DEBUG [main] zookeeper.ZKUtil(120): split-log-worker-tests opening connection to ZooKeeper with ensemble (localhost:52668) 2012-12-28 18:00:09,023 INFO [main] zookeeper.RecoverableZooKeeper(104): The identifier of this process is 15724@acer.localdomain 2012-12-28 18:00:09,074 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(277): split-log-worker-tests Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2012-12-28 18:00:09,075 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(354): split-log-worker-tests-0x13be46344380000 connected 2012-12-28 18:00:09,104 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(277): split-log-worker-tests-0x13be463424e0000 Received ZooKeeper Event, type=None, state=Disconnected, path=null 2012-12-28 18:00:09,104 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(375): split-log-worker-tests-0x13be463424e0000 Received Disconnected from ZooKeeper, ignoring 2012-12-28 18:00:09,119 DEBUG [main] regionserver.TestSplitLogWorker(97): /hbase created 2012-12-28 18:00:09,163 DEBUG [main] regionserver.TestSplitLogWorker(100): /hbase/splitlog created 2012-12-28 18:00:09,164 INFO [main] regionserver.TestSplitLogWorker(261): testRescan 2012-12-28 18:00:09,165 INFO [SplitLogWorker-svr] regionserver.SplitLogWorker(138): SplitLogWorker svr starting 2012-12-28 18:00:09,318 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(277): split-log-worker-tests-0x13be46344380000 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitlog 2012-12-28 18:00:09,319 DEBUG [main-EventThread] regionserver.SplitLogWorker(493): tasks arrived or departed 2012-12-28 18:00:09,322 DEBUG [SplitLogWorker-svr] zookeeper.ZKUtil(1541): split-log-worker-tests-0x13be46344380000 Retrieved 18 byte(s) of data from znode /hbase/splitlog/task; data=unassigned manager 2012-12-28 18:00:09,363 INFO [SplitLogWorker-svr] regionserver.SplitLogWorker(257): worker svr acquired task /hbase/splitlog/task 2012-12-28 18:00:09,407 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(277): split-log-worker-tests-0x13be46344380000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitlog/task 2012-12-28 18:00:09,409 INFO [main-EventThread] regionserver.SplitLogWorker(415): task /hbase/splitlog/task preempted from svr, current task state and owner=unassigned manager 2012-12-28 18:00:09,410 INFO [main-EventThread] regionserver.SplitLogWorker(506): Sending interrupt to stop the worker thread 2012-12-28 18:00:09,410 WARN [SplitLogWorker-svr] regionserver.SplitLogWorker(282): task execution prempted /hbase/splitlog/task 2012-12-28 18:00:09,410 INFO [SplitLogWorker-svr] regionserver.SplitLogWorker(306): worker svr done with task /hbase/splitlog/task in 47ms 2012-12-28 18:00:09,463 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(277): split-log-worker-tests-0x13be46344380000 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitlog 2012-12-28 18:00:09,463 DEBUG [main-EventThread] regionserver.SplitLogWorker(493): tasks arrived or departed 2012-12-28 18:00:09,466 DEBUG [SplitLogWorker-svr] zookeeper.ZKUtil(1541): split-log-worker-tests-0x13be46344380000 Retrieved 18 byte(s) of data from znode /hbase/splitlog/task; data=unassigned manager 2012-12-28 18:00:09,508 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(277): split-log-worker-tests-0x13be46344380000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitlog/task 2012-12-28 18:00:09,508 INFO [SplitLogWorker-svr] regionserver.SplitLogWorker(257): worker svr acquired task /hbase/splitlog/task 2012-12-28 18:00:09,552 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(277): split-log-worker-tests-0x13be46344380000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitlog/task 2012-12-28 18:00:09,554 INFO [main-EventThread] regionserver.SplitLogWorker(415): task /hbase/splitlog/task preempted from svr, current task state and owner=unassigned manager 2012-12-28 18:00:09,554 INFO [main-EventThread] regionserver.SplitLogWorker(506): Sending interrupt to stop the worker thread 2012-12-28 18:00:09,554 WARN [SplitLogWorker-svr] regionserver.SplitLogWorker(282): task execution prempted /hbase/splitlog/task 2012-12-28 18:00:09,555 INFO [SplitLogWorker-svr] regionserver.SplitLogWorker(306): worker svr done with task /hbase/splitlog/task in 46ms 2012-12-28 18:00:09,556 DEBUG [SplitLogWorker-svr] zookeeper.ZKUtil(1541): split-log-worker-tests-0x13be46344380000 Retrieved 18 byte(s) of data from znode /hbase/splitlog/RESCAN15724@acer.localdomain0000000001; data=unassigned manager 2012-12-28 18:00:09,642 INFO [SplitLogWorker-svr] regionserver.SplitLogWorker(372): successfully transitioned task /hbase/splitlog/RESCAN15724@acer.localdomain0000000001 to final state done 2012-12-28 18:00:09,645 DEBUG [main] regionserver.TestSplitLogWorker(296): [RESCAN15724@acer.localdomain0000000001, task] 2012-12-28 18:00:09,647 DEBUG [main] zookeeper.ZKUtil(1541): split-log-worker-tests-0x13be46344380000 Retrieved 8 byte(s) of data from znode /hbase/splitlog/RESCAN15724@acer.localdomain0000000001; data=done svr 2012-12-28 18:00:09,650 INFO [main] zookeeper.MiniZooKeeperCluster(235): Shutdown MiniZK cluster with all ZK servers 2012-12-28 18:00:09,651 INFO [main] hbase.ResourceChecker(157): after regionserver.TestSplitLogWorker#testRescan: 19 threads (was 15), 108 file descriptors (was 103). 0 connections, -thread leak?- -file handle leak?- 2012-12-28 18:00:09,652 INFO [main] hbase.ResourceChecker(182): after regionserver.TestSplitLogWorker#testRescan: potentially hanging thread - SplitLogWorker-svr 2012-12-28 18:00:09,652 INFO [main] hbase.ResourceChecker(185): java.lang.Object.wait(Native Method) 2012-12-28 18:00:09,653 INFO [main] hbase.ResourceChecker(185): java.lang.Object.wait(Object.java:502) 2012-12-28 18:00:09,653 INFO [main] hbase.ResourceChecker(185): org.apache.hadoop.hbase.regionserver.SplitLogWorker.taskLoop(SplitLogWorker.java:203) 2012-12-28 18:00:09,653 INFO [main] hbase.ResourceChecker(185): org.apache.hadoop.hbase.regionserver.SplitLogWorker.run(SplitLogWorker.java:163) 2012-12-28 18:00:09,654 INFO [main] hbase.ResourceChecker(185): java.lang.Thread.run(Thread.java:679) 2012-12-28 18:00:09,654 INFO [main] hbase.ResourceChecker(182): after regionserver.TestSplitLogWorker#testRescan: potentially hanging thread - main-SendThread(localhost.localdomain:52668) 2012-12-28 18:00:09,654 INFO [main] hbase.ResourceChecker(185): java.lang.Thread.sleep(Native Method) 2012-12-28 18:00:09,655 INFO [main] hbase.ResourceChecker(185): org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:219) 2012-12-28 18:00:09,655 INFO [main] hbase.ResourceChecker(185): org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1157) 2012-12-28 18:00:09,656 INFO [main] hbase.ResourceChecker(185): org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1097)