2013-10-04 14:30:07,125 INFO [pool-1-thread-1] hbase.ResourceChecker(147): before: master.TestSplitLogManager#testTaskErr Thread=5, OpenFileDescriptor=90, MaxFileDescriptor=10240, SystemLoadAverage=1067, ProcessCount=264, AvailableMemoryMB=3510, ConnectionCount=0 2013-10-04 14:30:07,386 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(398): Created new mini-cluster data directory: /Users/stack/checkouts/hbase/hbase-server/target/test-data/beb7a788-857e-44ec-a61e-5b7716b9d10e/dfscluster_ab3cb1c2-fa9a-4b95-8c50-b02906925c9b, deleteOnExit=true 2013-10-04 14:30:07,648 INFO [pool-1-thread-1] zookeeper.MiniZooKeeperCluster(200): Started MiniZK Cluster and connect 1 ZK server on client port: 63473 2013-10-04 14:30:07,722 INFO [pool-1-thread-1] zookeeper.RecoverableZooKeeper(120): Process identifier=split-log-manager-testsd2183943-5c0a-4d0f-9494-5601022b66ca connecting to ZooKeeper ensemble=localhost:63473 2013-10-04 14:30:07,758 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd2183943-5c0a-4d0f-9494-5601022b66ca Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2013-10-04 14:30:07,766 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(387): split-log-manager-testsd2183943-5c0a-4d0f-9494-5601022b66ca-0x1418560ad2a0000 connected 2013-10-04 14:30:07,781 DEBUG [pool-1-thread-1] master.TestSplitLogManager(119): /hbase created 2013-10-04 14:30:07,786 DEBUG [pool-1-thread-1] master.TestSplitLogManager(122): /hbase/splitWAL created 2013-10-04 14:30:07,809 INFO [pool-1-thread-1] master.TestSplitLogManager(347): TestTaskErr - cleanup task node once in ERR state 2013-10-04 14:30:07,813 INFO [pool-1-thread-1] master.SplitLogManager(227): Timeout=6000, unassigned timeout=12000, distributedLogReplay=false 2013-10-04 14:30:07,816 INFO [pool-1-thread-1] master.SplitLogManager(1108): Found 0 orphan tasks and 0 rescan nodes 2013-10-04 14:30:07,820 DEBUG [pool-1-thread-1] zookeeper.ZKUtil(430): split-log-manager-testsd2183943-5c0a-4d0f-9494-5601022b66ca-0x1418560ad2a0000 Set watcher on znode that does not yet exist, /hbase/splitWAL/foo%2F1 2013-10-04 14:30:07,917 INFO [dummy-master,1,1.splitLogManagerTimeoutMonitor] master.SplitLogManager$TimeoutMonitor(1408): total tasks = 1 unassigned = 1 tasks={/hbase/splitWAL/foo%2F1=last_update = -1 last_version = -1 cur_worker_name = null status = in_progress incarnation = 0 resubmits = 0 batch = installed = 1 done = 0 error = 0} 2013-10-04 14:30:08,125 DEBUG [pool-1-thread-1] master.TestSplitLogManager(186): waiting for task node creation 2013-10-04 14:30:08,128 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd2183943-5c0a-4d0f-9494-5601022b66ca-0x1418560ad2a0000 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/splitWAL/foo%2F1 2013-10-04 14:30:08,128 DEBUG [pool-1-thread-1-EventThread] regionserver.TestMasterAddressManager$NodeCreationListener(107): nodeCreated(/hbase/splitWAL/foo%2F1) 2013-10-04 14:30:08,129 DEBUG [pool-1-thread-1] master.TestSplitLogManager(188): task created 2013-10-04 14:30:08,131 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(711): put up splitlog task at znode /hbase/splitWAL/foo%2F1 2013-10-04 14:30:08,141 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(772): task /hbase/splitWAL/foo%2F1 entered state: ERR worker1,1,1 2013-10-04 14:30:08,143 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(864): Skipping resubmissions of task /hbase/splitWAL/foo%2F1 because threshold 0 reached 2013-10-04 14:30:08,143 WARN [pool-1-thread-1-EventThread] master.SplitLogManager(677): Error splitting /hbase/splitWAL/foo%2F1 2013-10-04 14:30:08,147 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd2183943-5c0a-4d0f-9494-5601022b66ca-0x1418560ad2a0000 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/splitWAL/foo%2F1 2013-10-04 14:30:08,147 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager$DeleteAsyncCallback(1605): deleted /hbase/splitWAL/foo%2F1 2013-10-04 14:30:08,148 INFO [pool-1-thread-1] hbase.Waiter(174): Waiting up to [3,200] milli-secs(wait.for.ratio=[1]) 2013-10-04 14:30:08,149 INFO [dummy-master,1,1.splitLogManagerTimeoutMonitor] hbase.Chore(93): dummy-master,1,1.splitLogManagerTimeoutMonitor exiting 2013-10-04 14:30:08,151 INFO [pool-1-thread-1] zookeeper.MiniZooKeeperCluster(249): Shutdown MiniZK cluster with all ZK servers 2013-10-04 14:30:08,251 INFO [pool-1-thread-1] hbase.ResourceChecker(171): after: master.TestSplitLogManager#testTaskErr Thread=10 (was 5) Potentially hanging thread: AWT-AppKit Potentially hanging thread: SessionTracker java.lang.Object.wait(Native Method) org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:146) Potentially hanging thread: pool-1-thread-1-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491) Potentially hanging thread: AWT-Shutdown java.lang.Object.wait(Native Method) sun.awt.AWTAutoShutdown.run(AWTAutoShutdown.java:284) java.lang.Thread.run(Thread.java:680) Potentially hanging thread: pool-1-thread-1-SendThread(localhost:63473) java.lang.Thread.sleep(Native Method) org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:219) org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1157) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1097) - Thread LEAK? -, OpenFileDescriptor=98 (was 90) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=10240 (was 10240), SystemLoadAverage=1067 (was 1067), ProcessCount=264 (was 264), AvailableMemoryMB=3445 (was 3510), ConnectionCount=0 (was 0) 2013-10-04 14:30:08,251 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd2183943-5c0a-4d0f-9494-5601022b66ca-0x1418560ad2a0000 Received ZooKeeper Event, type=None, state=Disconnected, path=null 2013-10-04 14:30:08,252 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(392): split-log-manager-testsd2183943-5c0a-4d0f-9494-5601022b66ca-0x1418560ad2a0000 Received Disconnected from ZooKeeper, ignoring 2013-10-04 14:30:08,337 INFO [pool-1-thread-1] hbase.ResourceChecker(147): before: master.TestSplitLogManager#testMultipleResubmits Thread=10, OpenFileDescriptor=100, MaxFileDescriptor=10240, SystemLoadAverage=1067, ProcessCount=264, AvailableMemoryMB=3452, ConnectionCount=0 2013-10-04 14:30:08,370 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(341): System.getProperty("hadoop.log.dir") already set to: /Users/stack/checkouts/hbase/hbase-server/target/test-data/beb7a788-857e-44ec-a61e-5b7716b9d10e/hadoop-log-dir so I do NOT create it in target/test-data/da35303b-0a65-41a8-9fd8-e382c7a52aa3 2013-10-04 14:30:08,370 WARN [pool-1-thread-1] hbase.HBaseTestingUtility(345): hadoop.log.dir property value differs in configuration and system: Configuration=/Users/stack/checkouts/hbase/hbase-server/../logs while System=/Users/stack/checkouts/hbase/hbase-server/target/test-data/beb7a788-857e-44ec-a61e-5b7716b9d10e/hadoop-log-dir Erasing configuration value by system value. 2013-10-04 14:30:08,370 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(341): System.getProperty("hadoop.tmp.dir") already set to: /Users/stack/checkouts/hbase/hbase-server/target/test-data/beb7a788-857e-44ec-a61e-5b7716b9d10e/hadoop-tmp-dir so I do NOT create it in target/test-data/da35303b-0a65-41a8-9fd8-e382c7a52aa3 2013-10-04 14:30:08,371 WARN [pool-1-thread-1] hbase.HBaseTestingUtility(345): hadoop.tmp.dir property value differs in configuration and system: Configuration=/tmp/hadoop-stack while System=/Users/stack/checkouts/hbase/hbase-server/target/test-data/beb7a788-857e-44ec-a61e-5b7716b9d10e/hadoop-tmp-dir Erasing configuration value by system value. 2013-10-04 14:30:08,371 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(398): Created new mini-cluster data directory: /Users/stack/checkouts/hbase/hbase-server/target/test-data/da35303b-0a65-41a8-9fd8-e382c7a52aa3/dfscluster_ef03e137-e33c-413c-a85c-f6b81b5a40d6, deleteOnExit=true 2013-10-04 14:30:08,380 INFO [pool-1-thread-1] zookeeper.MiniZooKeeperCluster(200): Started MiniZK Cluster and connect 1 ZK server on client port: 61971 2013-10-04 14:30:08,383 INFO [pool-1-thread-1] zookeeper.RecoverableZooKeeper(120): Process identifier=split-log-manager-tests6f6dfca2-fe4e-4a5d-ae95-416b38a3fa2c connecting to ZooKeeper ensemble=localhost:61971 2013-10-04 14:30:08,389 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-tests6f6dfca2-fe4e-4a5d-ae95-416b38a3fa2c Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2013-10-04 14:30:08,390 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(387): split-log-manager-tests6f6dfca2-fe4e-4a5d-ae95-416b38a3fa2c-0x1418560b0760000 connected 2013-10-04 14:30:08,394 DEBUG [pool-1-thread-1] master.TestSplitLogManager(119): /hbase created 2013-10-04 14:30:08,399 DEBUG [pool-1-thread-1] master.TestSplitLogManager(122): /hbase/splitWAL created 2013-10-04 14:30:08,400 INFO [pool-1-thread-1] master.TestSplitLogManager(266): TestMultipleResbmits - no indefinite resubmissions 2013-10-04 14:30:08,400 INFO [pool-1-thread-1] master.SplitLogManager(227): Timeout=6000, unassigned timeout=12000, distributedLogReplay=false 2013-10-04 14:30:08,403 INFO [pool-1-thread-1] master.SplitLogManager(1108): Found 0 orphan tasks and 0 rescan nodes 2013-10-04 14:30:08,405 DEBUG [pool-1-thread-1] zookeeper.ZKUtil(430): split-log-manager-tests6f6dfca2-fe4e-4a5d-ae95-416b38a3fa2c-0x1418560b0760000 Set watcher on znode that does not yet exist, /hbase/splitWAL/foo%2F1 2013-10-04 14:30:08,406 DEBUG [pool-1-thread-1] master.TestSplitLogManager(186): waiting for task node creation 2013-10-04 14:30:08,409 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-tests6f6dfca2-fe4e-4a5d-ae95-416b38a3fa2c-0x1418560b0760000 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/splitWAL/foo%2F1 2013-10-04 14:30:08,409 DEBUG [pool-1-thread-1-EventThread] regionserver.TestMasterAddressManager$NodeCreationListener(107): nodeCreated(/hbase/splitWAL/foo%2F1) 2013-10-04 14:30:08,409 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(711): put up splitlog task at znode /hbase/splitWAL/foo%2F1 2013-10-04 14:30:08,410 DEBUG [pool-1-thread-1] master.TestSplitLogManager(188): task created 2013-10-04 14:30:08,413 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(753): task not yet acquired /hbase/splitWAL/foo%2F1 ver = 0 2013-10-04 14:30:08,417 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-tests6f6dfca2-fe4e-4a5d-ae95-416b38a3fa2c-0x1418560b0760000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/foo%2F1 2013-10-04 14:30:08,419 INFO [pool-1-thread-1] hbase.Waiter(174): Waiting up to [3,200] milli-secs(wait.for.ratio=[1]) 2013-10-04 14:30:08,421 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(826): task /hbase/splitWAL/foo%2F1 acquired by worker1,1,1 2013-10-04 14:30:08,430 INFO [pool-1-thread-1] hbase.Waiter(174): Waiting up to [9,600] milli-secs(wait.for.ratio=[1]) 2013-10-04 14:30:08,502 INFO [dummy-master,1,1.splitLogManagerTimeoutMonitor] master.SplitLogManager$TimeoutMonitor(1408): total tasks = 1 unassigned = 0 tasks={/hbase/splitWAL/foo%2F1=last_update = 1380922208421 last_version = 1 cur_worker_name = worker1,1,1 status = in_progress incarnation = 0 resubmits = 0 batch = installed = 1 done = 0 error = 0} 2013-10-04 14:30:13,556 INFO [dummy-master,1,1.splitLogManagerTimeoutMonitor] master.SplitLogManager$TimeoutMonitor(1408): total tasks = 1 unassigned = 0 tasks={/hbase/splitWAL/foo%2F1=last_update = 1380922208421 last_version = 1 cur_worker_name = worker1,1,1 status = in_progress incarnation = 0 resubmits = 0 batch = installed = 1 done = 0 error = 0} 2013-10-04 14:30:14,462 INFO [dummy-master,1,1.splitLogManagerTimeoutMonitor] master.SplitLogManager(875): resubmitting task /hbase/splitWAL/foo%2F1 2013-10-04 14:30:14,464 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-tests6f6dfca2-fe4e-4a5d-ae95-416b38a3fa2c-0x1418560b0760000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/foo%2F1 2013-10-04 14:30:14,466 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(753): task not yet acquired /hbase/splitWAL/foo%2F1 ver = 2 2013-10-04 14:30:14,466 INFO [dummy-master,1,1.splitLogManagerTimeoutMonitor] master.SplitLogManager$TimeoutMonitor(1412): resubmitted 1 out of 1 tasks 2013-10-04 14:30:14,469 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(761): task /hbase/splitWAL/RESCAN0000000001 entered state: DONE dummy-master,1,1 2013-10-04 14:30:14,471 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-tests6f6dfca2-fe4e-4a5d-ae95-416b38a3fa2c-0x1418560b0760000 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/splitWAL/RESCAN0000000001 2013-10-04 14:30:14,471 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager$DeleteAsyncCallback(1605): deleted /hbase/splitWAL/RESCAN0000000001 2013-10-04 14:30:14,471 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(943): deleted task without in memory state /hbase/splitWAL/RESCAN0000000001 2013-10-04 14:30:14,476 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-tests6f6dfca2-fe4e-4a5d-ae95-416b38a3fa2c-0x1418560b0760000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/foo%2F1 2013-10-04 14:30:14,477 INFO [pool-1-thread-1] hbase.Waiter(174): Waiting up to [3,200] milli-secs(wait.for.ratio=[1]) 2013-10-04 14:30:14,478 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(826): task /hbase/splitWAL/foo%2F1 acquired by worker2,1,1 2013-10-04 14:30:14,488 INFO [pool-1-thread-1] hbase.Waiter(174): Waiting up to [9,600] milli-secs(wait.for.ratio=[1]) 2013-10-04 14:30:18,599 INFO [dummy-master,1,1.splitLogManagerTimeoutMonitor] master.SplitLogManager$TimeoutMonitor(1408): total tasks = 1 unassigned = 0 tasks={/hbase/splitWAL/foo%2F1=last_update = 1380922214478 last_version = 3 cur_worker_name = worker2,1,1 status = in_progress incarnation = 1 resubmits = 1 batch = installed = 1 done = 0 error = 0} 2013-10-04 14:30:20,514 INFO [dummy-master,1,1.splitLogManagerTimeoutMonitor] master.SplitLogManager(875): resubmitting task /hbase/splitWAL/foo%2F1 2013-10-04 14:30:20,515 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-tests6f6dfca2-fe4e-4a5d-ae95-416b38a3fa2c-0x1418560b0760000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/foo%2F1 2013-10-04 14:30:20,515 INFO [dummy-master,1,1.splitLogManagerTimeoutMonitor] master.SplitLogManager$TimeoutMonitor(1412): resubmitted 1 out of 1 tasks 2013-10-04 14:30:20,517 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(753): task not yet acquired /hbase/splitWAL/foo%2F1 ver = 4 2013-10-04 14:30:20,518 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(761): task /hbase/splitWAL/RESCAN0000000002 entered state: DONE dummy-master,1,1 2013-10-04 14:30:20,522 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-tests6f6dfca2-fe4e-4a5d-ae95-416b38a3fa2c-0x1418560b0760000 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/splitWAL/RESCAN0000000002 2013-10-04 14:30:20,522 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager$DeleteAsyncCallback(1605): deleted /hbase/splitWAL/RESCAN0000000002 2013-10-04 14:30:20,522 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(943): deleted task without in memory state /hbase/splitWAL/RESCAN0000000002 2013-10-04 14:30:20,522 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-tests6f6dfca2-fe4e-4a5d-ae95-416b38a3fa2c-0x1418560b0760000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/foo%2F1 2013-10-04 14:30:20,523 INFO [pool-1-thread-1] hbase.Waiter(174): Waiting up to [3,200] milli-secs(wait.for.ratio=[1]) 2013-10-04 14:30:20,524 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(826): task /hbase/splitWAL/foo%2F1 acquired by worker3,1,1 2013-10-04 14:30:20,534 INFO [pool-1-thread-1] hbase.Waiter(174): Waiting up to [9,600] milli-secs(wait.for.ratio=[1]) 2013-10-04 14:30:23,639 INFO [dummy-master,1,1.splitLogManagerTimeoutMonitor] master.SplitLogManager$TimeoutMonitor(1408): total tasks = 1 unassigned = 0 tasks={/hbase/splitWAL/foo%2F1=last_update = 1380922220524 last_version = 5 cur_worker_name = worker3,1,1 status = in_progress incarnation = 2 resubmits = 2 batch = installed = 1 done = 0 error = 0} 2013-10-04 14:30:26,574 INFO [dummy-master,1,1.splitLogManagerTimeoutMonitor] master.SplitLogManager(864): Skipping resubmissions of task /hbase/splitWAL/foo%2F1 because threshold 2 reached 2013-10-04 14:30:28,691 INFO [dummy-master,1,1.splitLogManagerTimeoutMonitor] master.SplitLogManager$TimeoutMonitor(1408): total tasks = 1 unassigned = 0 tasks={/hbase/splitWAL/foo%2F1=last_update = 1380922220524 last_version = 5 cur_worker_name = worker3,1,1 status = in_progress incarnation = 2 resubmits = 2 batch = installed = 1 done = 0 error = 0} 2013-10-04 14:30:33,728 INFO [dummy-master,1,1.splitLogManagerTimeoutMonitor] master.SplitLogManager$TimeoutMonitor(1408): total tasks = 1 unassigned = 0 tasks={/hbase/splitWAL/foo%2F1=last_update = 1380922220524 last_version = 5 cur_worker_name = worker3,1,1 status = in_progress incarnation = 2 resubmits = 2 batch = installed = 1 done = 0 error = 0} 2013-10-04 14:30:36,181 INFO [dummy-master,1,1.splitLogManagerTimeoutMonitor] hbase.Chore(93): dummy-master,1,1.splitLogManagerTimeoutMonitor exiting 2013-10-04 14:30:36,183 INFO [pool-1-thread-1] zookeeper.MiniZooKeeperCluster(249): Shutdown MiniZK cluster with all ZK servers 2013-10-04 14:30:36,283 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-tests6f6dfca2-fe4e-4a5d-ae95-416b38a3fa2c-0x1418560b0760000 Received ZooKeeper Event, type=None, state=Disconnected, path=null 2013-10-04 14:30:36,284 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(392): split-log-manager-tests6f6dfca2-fe4e-4a5d-ae95-416b38a3fa2c-0x1418560b0760000 Received Disconnected from ZooKeeper, ignoring 2013-10-04 14:30:36,327 INFO [pool-1-thread-1] hbase.ResourceChecker(171): after: master.TestSplitLogManager#testMultipleResubmits Thread=11 (was 10) Potentially hanging thread: pool-1-thread-1-SendThread(localhost:61971) java.lang.Thread.sleep(Native Method) org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:219) org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1157) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1097) - Thread LEAK? -, OpenFileDescriptor=106 (was 100) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=10240 (was 10240), SystemLoadAverage=891 (was 1067), ProcessCount=259 (was 264), AvailableMemoryMB=3549 (was 3452) - AvailableMemoryMB LEAK? -, ConnectionCount=0 (was 0) 2013-10-04 14:30:36,468 INFO [pool-1-thread-1] hbase.ResourceChecker(147): before: master.TestSplitLogManager#testTaskResigned Thread=11, OpenFileDescriptor=108, MaxFileDescriptor=10240, SystemLoadAverage=891, ProcessCount=260, AvailableMemoryMB=3550, ConnectionCount=0 2013-10-04 14:30:36,512 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(341): System.getProperty("hadoop.log.dir") already set to: /Users/stack/checkouts/hbase/hbase-server/target/test-data/beb7a788-857e-44ec-a61e-5b7716b9d10e/hadoop-log-dir so I do NOT create it in target/test-data/54fd974a-2423-4c51-aeb0-be0b05c1a3b0 2013-10-04 14:30:36,513 WARN [pool-1-thread-1] hbase.HBaseTestingUtility(345): hadoop.log.dir property value differs in configuration and system: Configuration=/Users/stack/checkouts/hbase/hbase-server/../logs while System=/Users/stack/checkouts/hbase/hbase-server/target/test-data/beb7a788-857e-44ec-a61e-5b7716b9d10e/hadoop-log-dir Erasing configuration value by system value. 2013-10-04 14:30:36,513 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(341): System.getProperty("hadoop.tmp.dir") already set to: /Users/stack/checkouts/hbase/hbase-server/target/test-data/beb7a788-857e-44ec-a61e-5b7716b9d10e/hadoop-tmp-dir so I do NOT create it in target/test-data/54fd974a-2423-4c51-aeb0-be0b05c1a3b0 2013-10-04 14:30:36,513 WARN [pool-1-thread-1] hbase.HBaseTestingUtility(345): hadoop.tmp.dir property value differs in configuration and system: Configuration=/tmp/hadoop-stack while System=/Users/stack/checkouts/hbase/hbase-server/target/test-data/beb7a788-857e-44ec-a61e-5b7716b9d10e/hadoop-tmp-dir Erasing configuration value by system value. 2013-10-04 14:30:36,514 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(398): Created new mini-cluster data directory: /Users/stack/checkouts/hbase/hbase-server/target/test-data/54fd974a-2423-4c51-aeb0-be0b05c1a3b0/dfscluster_edbb2329-181a-483d-b3f8-15e44d139851, deleteOnExit=true 2013-10-04 14:30:36,528 INFO [pool-1-thread-1] zookeeper.MiniZooKeeperCluster(200): Started MiniZK Cluster and connect 1 ZK server on client port: 59585 2013-10-04 14:30:36,532 INFO [pool-1-thread-1] zookeeper.RecoverableZooKeeper(120): Process identifier=split-log-manager-tests87be0124-b372-4a6d-a3e5-73e3c5a59e5c connecting to ZooKeeper ensemble=localhost:59585 2013-10-04 14:30:36,540 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-tests87be0124-b372-4a6d-a3e5-73e3c5a59e5c Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2013-10-04 14:30:36,543 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(387): split-log-manager-tests87be0124-b372-4a6d-a3e5-73e3c5a59e5c-0x14185611e660000 connected 2013-10-04 14:30:36,553 DEBUG [pool-1-thread-1] master.TestSplitLogManager(119): /hbase created 2013-10-04 14:30:36,561 DEBUG [pool-1-thread-1] master.TestSplitLogManager(122): /hbase/splitWAL created 2013-10-04 14:30:36,562 INFO [pool-1-thread-1] master.TestSplitLogManager(370): TestTaskResigned - resubmit task node once in RESIGNED state 2013-10-04 14:30:36,562 INFO [pool-1-thread-1] master.SplitLogManager(227): Timeout=6000, unassigned timeout=12000, distributedLogReplay=false 2013-10-04 14:30:36,564 INFO [pool-1-thread-1] master.SplitLogManager(1108): Found 0 orphan tasks and 0 rescan nodes 2013-10-04 14:30:36,566 DEBUG [pool-1-thread-1] zookeeper.ZKUtil(430): split-log-manager-tests87be0124-b372-4a6d-a3e5-73e3c5a59e5c-0x14185611e660000 Set watcher on znode that does not yet exist, /hbase/splitWAL/foo%2F1 2013-10-04 14:30:36,566 DEBUG [pool-1-thread-1] master.TestSplitLogManager(186): waiting for task node creation 2013-10-04 14:30:36,568 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-tests87be0124-b372-4a6d-a3e5-73e3c5a59e5c-0x14185611e660000 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/splitWAL/foo%2F1 2013-10-04 14:30:36,569 DEBUG [pool-1-thread-1-EventThread] regionserver.TestMasterAddressManager$NodeCreationListener(107): nodeCreated(/hbase/splitWAL/foo%2F1) 2013-10-04 14:30:36,569 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(711): put up splitlog task at znode /hbase/splitWAL/foo%2F1 2013-10-04 14:30:36,569 DEBUG [pool-1-thread-1] master.TestSplitLogManager(188): task created 2013-10-04 14:30:36,571 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(753): task not yet acquired /hbase/splitWAL/foo%2F1 ver = 0 2013-10-04 14:30:36,572 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-tests87be0124-b372-4a6d-a3e5-73e3c5a59e5c-0x14185611e660000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/foo%2F1 2013-10-04 14:30:36,573 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(758): task /hbase/splitWAL/foo%2F1 entered state: RESIGNED worker1,1,1 2013-10-04 14:30:36,574 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(875): resubmitting task /hbase/splitWAL/foo%2F1 2013-10-04 14:30:36,577 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-tests87be0124-b372-4a6d-a3e5-73e3c5a59e5c-0x14185611e660000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/foo%2F1 2013-10-04 14:30:36,581 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(753): task not yet acquired /hbase/splitWAL/foo%2F1 ver = 2 2013-10-04 14:30:36,582 INFO [dummy-master,1,1.splitLogManagerTimeoutMonitor] hbase.Chore(93): dummy-master,1,1.splitLogManagerTimeoutMonitor exiting 2013-10-04 14:30:36,583 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(761): task /hbase/splitWAL/RESCAN0000000001 entered state: DONE dummy-master,1,1 2013-10-04 14:30:36,589 INFO [pool-1-thread-1] zookeeper.MiniZooKeeperCluster(249): Shutdown MiniZK cluster with all ZK servers 2013-10-04 14:30:36,687 WARN [pool-1-thread-1-EventThread] master.SplitLogManager$DeleteAsyncCallback(1588): delete rc=CONNECTIONLOSS for /hbase/splitWAL/RESCAN0000000001 remaining retries=3 2013-10-04 14:30:36,688 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-tests87be0124-b372-4a6d-a3e5-73e3c5a59e5c-0x14185611e660000 Received ZooKeeper Event, type=None, state=Disconnected, path=null 2013-10-04 14:30:36,688 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(392): split-log-manager-tests87be0124-b372-4a6d-a3e5-73e3c5a59e5c-0x14185611e660000 Received Disconnected from ZooKeeper, ignoring 2013-10-04 14:30:36,712 INFO [pool-1-thread-1] hbase.ResourceChecker(171): after: master.TestSplitLogManager#testTaskResigned Thread=14 (was 11) Potentially hanging thread: pool-1-thread-1-SendThread(localhost:59585) java.lang.Thread.sleep(Native Method) org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:219) org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1157) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1097) - Thread LEAK? -, OpenFileDescriptor=106 (was 108), MaxFileDescriptor=10240 (was 10240), SystemLoadAverage=891 (was 891), ProcessCount=259 (was 260), AvailableMemoryMB=3543 (was 3550), ConnectionCount=0 (was 0) 2013-10-04 14:30:38,216 WARN [pool-1-thread-1-EventThread] master.SplitLogManager$DeleteAsyncCallback(1588): delete rc=CONNECTIONLOSS for /hbase/splitWAL/RESCAN0000000001 remaining retries=2 2013-10-04 14:30:39,951 WARN [pool-1-thread-1-EventThread] master.SplitLogManager$DeleteAsyncCallback(1588): delete rc=CONNECTIONLOSS for /hbase/splitWAL/RESCAN0000000001 remaining retries=1 2013-10-04 14:30:41,659 WARN [pool-1-thread-1-EventThread] master.SplitLogManager$DeleteAsyncCallback(1588): delete rc=CONNECTIONLOSS for /hbase/splitWAL/RESCAN0000000001 remaining retries=0 2013-10-04 14:30:41,660 WARN [pool-1-thread-1-EventThread] master.SplitLogManager$DeleteAsyncCallback(1591): delete failed /hbase/splitWAL/RESCAN0000000001 2013-10-04 14:30:41,660 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(954): Failed to delete node /hbase/splitWAL/RESCAN0000000001 and will retry soon. 2013-10-04 14:30:36,848 INFO [pool-1-thread-1] hbase.ResourceChecker(147): before: master.TestSplitLogManager#testVanishingTaskZNode Thread=14, OpenFileDescriptor=108, MaxFileDescriptor=10240, SystemLoadAverage=891, ProcessCount=259, AvailableMemoryMB=3541, ConnectionCount=0 2013-10-04 14:30:36,872 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(341): System.getProperty("hadoop.log.dir") already set to: /Users/stack/checkouts/hbase/hbase-server/target/test-data/beb7a788-857e-44ec-a61e-5b7716b9d10e/hadoop-log-dir so I do NOT create it in target/test-data/1b1c3b11-46ed-4320-b05d-9256644ea669 2013-10-04 14:30:36,873 WARN [pool-1-thread-1] hbase.HBaseTestingUtility(345): hadoop.log.dir property value differs in configuration and system: Configuration=/Users/stack/checkouts/hbase/hbase-server/../logs while System=/Users/stack/checkouts/hbase/hbase-server/target/test-data/beb7a788-857e-44ec-a61e-5b7716b9d10e/hadoop-log-dir Erasing configuration value by system value. 2013-10-04 14:30:36,873 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(341): System.getProperty("hadoop.tmp.dir") already set to: /Users/stack/checkouts/hbase/hbase-server/target/test-data/beb7a788-857e-44ec-a61e-5b7716b9d10e/hadoop-tmp-dir so I do NOT create it in target/test-data/1b1c3b11-46ed-4320-b05d-9256644ea669 2013-10-04 14:30:36,873 WARN [pool-1-thread-1] hbase.HBaseTestingUtility(345): hadoop.tmp.dir property value differs in configuration and system: Configuration=/tmp/hadoop-stack while System=/Users/stack/checkouts/hbase/hbase-server/target/test-data/beb7a788-857e-44ec-a61e-5b7716b9d10e/hadoop-tmp-dir Erasing configuration value by system value. 2013-10-04 14:30:36,874 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(398): Created new mini-cluster data directory: /Users/stack/checkouts/hbase/hbase-server/target/test-data/1b1c3b11-46ed-4320-b05d-9256644ea669/dfscluster_9f978c08-f76e-4488-9e31-9275544c8e96, deleteOnExit=true 2013-10-04 14:30:36,889 INFO [pool-1-thread-1] zookeeper.MiniZooKeeperCluster(200): Started MiniZK Cluster and connect 1 ZK server on client port: 53996 2013-10-04 14:30:36,894 INFO [pool-1-thread-1] zookeeper.RecoverableZooKeeper(120): Process identifier=split-log-manager-testsd52c73a2-038f-4d4d-adc7-4f87b51300a4 connecting to ZooKeeper ensemble=localhost:53996 2013-10-04 14:30:36,899 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd52c73a2-038f-4d4d-adc7-4f87b51300a4 Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2013-10-04 14:30:36,903 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(387): split-log-manager-testsd52c73a2-038f-4d4d-adc7-4f87b51300a4-0x14185611fd00000 connected 2013-10-04 14:30:36,913 DEBUG [pool-1-thread-1] master.TestSplitLogManager(119): /hbase created 2013-10-04 14:30:36,919 DEBUG [pool-1-thread-1] master.TestSplitLogManager(122): /hbase/splitWAL created 2013-10-04 14:30:36,921 INFO [Thread-10] master.TestSplitLogManager(496): testVanishingTaskZNode 2013-10-04 14:30:36,922 INFO [Thread-10] master.SplitLogManager(227): Timeout=6000, unassigned timeout=0, distributedLogReplay=false 2013-10-04 14:30:36,925 INFO [Thread-10] master.SplitLogManager(1108): Found 0 orphan tasks and 0 rescan nodes 2013-10-04 14:30:36,930 DEBUG [Thread-10] fs.HFileSystem(213): The file system is not a DistributedFileSystem. Skipping on block location reordering 2013-10-04 14:30:36,947 WARN [Thread-10] util.NativeCodeLoader(52): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2013-10-04 14:30:36,955 INFO [Thread-10] hbase.Waiter(174): Waiting up to [10,000] milli-secs(wait.for.ratio=[1]) 2013-10-04 14:30:36,958 WARN [Thread-11] wal.HLogUtil(189): Invalid log file path=file:/Users/stack/checkouts/hbase/hbase-server/3e294b1d-c42a-4417-9017-9317ac1f3cf9 java.lang.IllegalArgumentException: Not a host:port pair: hbase-server at org.apache.hadoop.hbase.util.Addressing.parseHostname(Addressing.java:62) at org.apache.hadoop.hbase.ServerName.(ServerName.java:125) at org.apache.hadoop.hbase.ServerName.parseServerName(ServerName.java:318) at org.apache.hadoop.hbase.regionserver.wal.HLogUtil.getServerNameFromHLogDirectoryName(HLogUtil.java:186) at org.apache.hadoop.hbase.master.SplitLogManager.splitLogDistributed(SplitLogManager.java:298) at org.apache.hadoop.hbase.master.SplitLogManager.splitLogDistributed(SplitLogManager.java:278) at org.apache.hadoop.hbase.master.TestSplitLogManager$5.run(TestSplitLogManager.java:515) 2013-10-04 14:30:37,270 DEBUG [Thread-11] master.SplitLogManager(327): Scheduling batch of logs to split 2013-10-04 14:30:37,270 INFO [Thread-11] master.SplitLogManager(329): started splitting 1 logs in [file:/Users/stack/checkouts/hbase/hbase-server/3e294b1d-c42a-4417-9017-9317ac1f3cf9] 2013-10-04 14:30:37,274 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(711): put up splitlog task at znode /hbase/splitWAL/file%3A%2FUsers%2Fstack%2Fcheckouts%2Fhbase%2Fhbase-server%2F3e294b1d-c42a-4417-9017-9317ac1f3cf9%2F50e6429b-9444-46f3-91c1-c6878aaad2de 2013-10-04 14:30:37,277 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(753): task not yet acquired /hbase/splitWAL/file%3A%2FUsers%2Fstack%2Fcheckouts%2Fhbase%2Fhbase-server%2F3e294b1d-c42a-4417-9017-9317ac1f3cf9%2F50e6429b-9444-46f3-91c1-c6878aaad2de ver = 0 2013-10-04 14:30:37,277 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd52c73a2-038f-4d4d-adc7-4f87b51300a4-0x14185611fd00000 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/splitWAL/file%3A%2FUsers%2Fstack%2Fcheckouts%2Fhbase%2Fhbase-server%2F3e294b1d-c42a-4417-9017-9317ac1f3cf9%2F50e6429b-9444-46f3-91c1-c6878aaad2de 2013-10-04 14:30:37,278 INFO [Thread-10] hbase.Waiter(174): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2013-10-04 14:30:37,378 WARN [Thread-11] master.SplitLogManager(427): Expected at least1 tasks in ZK, but actually there are 0 2013-10-04 14:30:37,378 WARN [Thread-11] master.SplitLogManager(431): No more task remaining (ZK or task map), splitting should have completed. Remaining tasks in ZK 0, active tasks in map 1 2013-10-04 14:30:37,481 WARN [Thread-11] master.SplitLogManager(427): Expected at least1 tasks in ZK, but actually there are 0 2013-10-04 14:30:37,482 WARN [Thread-11] master.SplitLogManager(431): No more task remaining (ZK or task map), splitting should have completed. Remaining tasks in ZK 0, active tasks in map 1 2013-10-04 14:30:37,584 WARN [Thread-11] master.SplitLogManager(427): Expected at least1 tasks in ZK, but actually there are 0 2013-10-04 14:30:37,586 WARN [Thread-11] master.SplitLogManager(431): No more task remaining (ZK or task map), splitting should have completed. Remaining tasks in ZK 0, active tasks in map 1 2013-10-04 14:30:37,691 WARN [Thread-11] master.SplitLogManager(427): Expected at least1 tasks in ZK, but actually there are 0 2013-10-04 14:30:37,691 WARN [Thread-11] master.SplitLogManager(431): No more task remaining (ZK or task map), splitting should have completed. Remaining tasks in ZK 0, active tasks in map 1 2013-10-04 14:30:37,794 WARN [Thread-11] master.SplitLogManager(427): Expected at least1 tasks in ZK, but actually there are 0 2013-10-04 14:30:37,794 WARN [Thread-11] master.SplitLogManager(431): No more task remaining (ZK or task map), splitting should have completed. Remaining tasks in ZK 0, active tasks in map 1 2013-10-04 14:30:37,899 WARN [Thread-11] master.SplitLogManager(427): Expected at least1 tasks in ZK, but actually there are 0 2013-10-04 14:30:37,899 WARN [Thread-11] master.SplitLogManager(431): No more task remaining (ZK or task map), splitting should have completed. Remaining tasks in ZK 0, active tasks in map 1 2013-10-04 14:30:37,924 INFO [dummy-master,1,1.splitLogManagerTimeoutMonitor] master.SplitLogManager$TimeoutMonitor(1408): total tasks = 1 unassigned = 1 tasks={/hbase/splitWAL/file%3A%2FUsers%2Fstack%2Fcheckouts%2Fhbase%2Fhbase-server%2F3e294b1d-c42a-4417-9017-9317ac1f3cf9%2F50e6429b-9444-46f3-91c1-c6878aaad2de=last_update = -1 last_version = -1 cur_worker_name = null status = in_progress incarnation = 0 resubmits = 0 batch = installed = 1 done = 0 error = 0} 2013-10-04 14:30:37,924 DEBUG [dummy-master,1,1.splitLogManagerTimeoutMonitor] master.SplitLogManager$TimeoutMonitor(1440): resubmitting unassigned task(s) after timeout 2013-10-04 14:30:37,931 WARN [pool-1-thread-1-EventThread] master.SplitLogManager$GetDataAsyncCallback(1535): task znode /hbase/splitWAL/file%3A%2FUsers%2Fstack%2Fcheckouts%2Fhbase%2Fhbase-server%2F3e294b1d-c42a-4417-9017-9317ac1f3cf9%2F50e6429b-9444-46f3-91c1-c6878aaad2de vanished. 2013-10-04 14:30:37,932 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(674): Done splitting /hbase/splitWAL/file%3A%2FUsers%2Fstack%2Fcheckouts%2Fhbase%2Fhbase-server%2F3e294b1d-c42a-4417-9017-9317ac1f3cf9%2F50e6429b-9444-46f3-91c1-c6878aaad2de 2013-10-04 14:30:37,936 INFO [Thread-10] hbase.Waiter(174): Waiting up to [3,200] milli-secs(wait.for.ratio=[1]) 2013-10-04 14:30:37,937 INFO [pool-1-thread-1-EventThread] master.SplitLogManager$DeleteAsyncCallback(1599): /hbase/splitWAL/file%3A%2FUsers%2Fstack%2Fcheckouts%2Fhbase%2Fhbase-server%2F3e294b1d-c42a-4417-9017-9317ac1f3cf9%2F50e6429b-9444-46f3-91c1-c6878aaad2de does not exist. Either was created but deleted behind our back by another pending delete OR was deleted in earlier retry rounds. zkretries = 3 2013-10-04 14:30:37,938 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(761): task /hbase/splitWAL/RESCAN0000000001 entered state: DONE dummy-master,1,1 2013-10-04 14:30:37,938 WARN [Thread-11] master.SplitLogManager(374): returning success without actually splitting and deleting all the log files in path file:/Users/stack/checkouts/hbase/hbase-server/3e294b1d-c42a-4417-9017-9317ac1f3cf9 2013-10-04 14:30:37,939 INFO [Thread-11] master.SplitLogManager(386): finished splitting (more than or equal to) 0 bytes in 1 log files in [file:/Users/stack/checkouts/hbase/hbase-server/3e294b1d-c42a-4417-9017-9317ac1f3cf9] in 669ms 2013-10-04 14:30:37,940 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd52c73a2-038f-4d4d-adc7-4f87b51300a4-0x14185611fd00000 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/splitWAL/RESCAN0000000001 2013-10-04 14:30:37,941 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager$DeleteAsyncCallback(1605): deleted /hbase/splitWAL/RESCAN0000000001 2013-10-04 14:30:37,941 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(943): deleted task without in memory state /hbase/splitWAL/RESCAN0000000001 2013-10-04 14:30:37,953 INFO [dummy-master,1,1.splitLogManagerTimeoutMonitor] hbase.Chore(93): dummy-master,1,1.splitLogManagerTimeoutMonitor exiting 2013-10-04 14:30:37,959 INFO [pool-1-thread-1] zookeeper.MiniZooKeeperCluster(249): Shutdown MiniZK cluster with all ZK servers 2013-10-04 14:30:38,056 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd52c73a2-038f-4d4d-adc7-4f87b51300a4-0x14185611fd00000 Received ZooKeeper Event, type=None, state=Disconnected, path=null 2013-10-04 14:30:38,057 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(392): split-log-manager-testsd52c73a2-038f-4d4d-adc7-4f87b51300a4-0x14185611fd00000 Received Disconnected from ZooKeeper, ignoring 2013-10-04 14:30:38,154 INFO [pool-1-thread-1] hbase.ResourceChecker(171): after: master.TestSplitLogManager#testVanishingTaskZNode Thread=17 (was 14) Potentially hanging thread: pool-1-thread-1-SendThread(localhost:53996) java.lang.Thread.sleep(Native Method) org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:219) org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1157) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1097) - Thread LEAK? -, OpenFileDescriptor=110 (was 108) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=10240 (was 10240), SystemLoadAverage=964 (was 891) - SystemLoadAverage LEAK? -, ProcessCount=259 (was 259), AvailableMemoryMB=3503 (was 3541), ConnectionCount=0 (was 0) 2013-10-04 14:30:38,319 INFO [pool-1-thread-1] hbase.ResourceChecker(147): before: master.TestSplitLogManager#testUnassignedOrphan Thread=14, OpenFileDescriptor=112, MaxFileDescriptor=10240, SystemLoadAverage=964, ProcessCount=260, AvailableMemoryMB=3505, ConnectionCount=0 2013-10-04 14:30:38,348 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(341): System.getProperty("hadoop.log.dir") already set to: /Users/stack/checkouts/hbase/hbase-server/target/test-data/beb7a788-857e-44ec-a61e-5b7716b9d10e/hadoop-log-dir so I do NOT create it in target/test-data/6dc7c74d-3439-432c-900d-74145abc5daa 2013-10-04 14:30:38,348 WARN [pool-1-thread-1] hbase.HBaseTestingUtility(345): hadoop.log.dir property value differs in configuration and system: Configuration=/Users/stack/checkouts/hbase/hbase-server/../logs while System=/Users/stack/checkouts/hbase/hbase-server/target/test-data/beb7a788-857e-44ec-a61e-5b7716b9d10e/hadoop-log-dir Erasing configuration value by system value. 2013-10-04 14:30:38,348 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(341): System.getProperty("hadoop.tmp.dir") already set to: /Users/stack/checkouts/hbase/hbase-server/target/test-data/beb7a788-857e-44ec-a61e-5b7716b9d10e/hadoop-tmp-dir so I do NOT create it in target/test-data/6dc7c74d-3439-432c-900d-74145abc5daa 2013-10-04 14:30:38,349 WARN [pool-1-thread-1] hbase.HBaseTestingUtility(345): hadoop.tmp.dir property value differs in configuration and system: Configuration=/tmp/hadoop-stack while System=/Users/stack/checkouts/hbase/hbase-server/target/test-data/beb7a788-857e-44ec-a61e-5b7716b9d10e/hadoop-tmp-dir Erasing configuration value by system value. 2013-10-04 14:30:38,349 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(398): Created new mini-cluster data directory: /Users/stack/checkouts/hbase/hbase-server/target/test-data/6dc7c74d-3439-432c-900d-74145abc5daa/dfscluster_b63a0519-6c7e-4c38-98f9-15386d256c5a, deleteOnExit=true 2013-10-04 14:30:38,378 INFO [pool-1-thread-1] zookeeper.MiniZooKeeperCluster(200): Started MiniZK Cluster and connect 1 ZK server on client port: 65267 2013-10-04 14:30:38,381 INFO [pool-1-thread-1] zookeeper.RecoverableZooKeeper(120): Process identifier=split-log-manager-tests740ad96a-471c-4368-bba0-251ac66d1432 connecting to ZooKeeper ensemble=localhost:65267 2013-10-04 14:30:38,385 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-tests740ad96a-471c-4368-bba0-251ac66d1432 Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2013-10-04 14:30:38,401 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(387): split-log-manager-tests740ad96a-471c-4368-bba0-251ac66d1432-0x141856125920000 connected 2013-10-04 14:30:38,424 DEBUG [pool-1-thread-1] master.TestSplitLogManager(119): /hbase created 2013-10-04 14:30:38,431 DEBUG [pool-1-thread-1] master.TestSplitLogManager(122): /hbase/splitWAL created 2013-10-04 14:30:38,447 INFO [pool-1-thread-1] master.TestSplitLogManager(237): TestUnassignedOrphan - an unassigned task is resubmitted at startup 2013-10-04 14:30:38,456 INFO [pool-1-thread-1] master.SplitLogManager(227): Timeout=6000, unassigned timeout=12000, distributedLogReplay=false 2013-10-04 14:30:38,461 INFO [pool-1-thread-1] master.SplitLogManager(1104): found orphan task orphan%2Ftest%2Fslash 2013-10-04 14:30:38,470 INFO [pool-1-thread-1] master.SplitLogManager(1108): Found 1 orphan tasks and 0 rescan nodes 2013-10-04 14:30:38,470 INFO [pool-1-thread-1] hbase.Waiter(174): Waiting up to [3,200] milli-secs(wait.for.ratio=[1]) 2013-10-04 14:30:38,471 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(753): task not yet acquired /hbase/splitWAL/orphan%2Ftest%2Fslash ver = 0 2013-10-04 14:30:38,472 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(1058): creating orphan task /hbase/splitWAL/orphan%2Ftest%2Fslash 2013-10-04 14:30:38,472 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(800): resubmitting unassigned orphan task /hbase/splitWAL/orphan%2Ftest%2Fslash 2013-10-04 14:30:38,472 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(875): resubmitting task /hbase/splitWAL/orphan%2Ftest%2Fslash 2013-10-04 14:30:38,475 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-tests740ad96a-471c-4368-bba0-251ac66d1432-0x141856125920000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2Ftest%2Fslash 2013-10-04 14:30:38,478 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(753): task not yet acquired /hbase/splitWAL/orphan%2Ftest%2Fslash ver = 1 2013-10-04 14:30:38,479 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(761): task /hbase/splitWAL/RESCAN0000000001 entered state: DONE dummy-master,1,1 2013-10-04 14:30:38,481 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-tests740ad96a-471c-4368-bba0-251ac66d1432-0x141856125920000 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/splitWAL/RESCAN0000000001 2013-10-04 14:30:38,482 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager$DeleteAsyncCallback(1605): deleted /hbase/splitWAL/RESCAN0000000001 2013-10-04 14:30:38,482 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(943): deleted task without in memory state /hbase/splitWAL/RESCAN0000000001 2013-10-04 14:30:38,483 INFO [pool-1-thread-1] hbase.Waiter(174): Waiting up to [3,200] milli-secs(wait.for.ratio=[1]) 2013-10-04 14:30:38,483 DEBUG [pool-1-thread-1] master.TestSplitLogManager(255): task = last_update = 1380922238475 last_version = -1 cur_worker_name = null status = in_progress incarnation = 1 resubmits = 0 batch = null 2013-10-04 14:30:38,490 INFO [dummy-master,1,1.splitLogManagerTimeoutMonitor] hbase.Chore(93): dummy-master,1,1.splitLogManagerTimeoutMonitor exiting 2013-10-04 14:30:38,499 INFO [pool-1-thread-1] zookeeper.MiniZooKeeperCluster(249): Shutdown MiniZK cluster with all ZK servers 2013-10-04 14:30:38,595 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-tests740ad96a-471c-4368-bba0-251ac66d1432-0x141856125920000 Received ZooKeeper Event, type=None, state=Disconnected, path=null 2013-10-04 14:30:38,595 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(392): split-log-manager-tests740ad96a-471c-4368-bba0-251ac66d1432-0x141856125920000 Received Disconnected from ZooKeeper, ignoring 2013-10-04 14:30:38,663 INFO [pool-1-thread-1] hbase.ResourceChecker(171): after: master.TestSplitLogManager#testUnassignedOrphan Thread=17 (was 14) Potentially hanging thread: pool-1-thread-1-SendThread(localhost:65267) java.lang.Thread.sleep(Native Method) org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:219) org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1157) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1097) - Thread LEAK? -, OpenFileDescriptor=118 (was 112) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=10240 (was 10240), SystemLoadAverage=964 (was 964), ProcessCount=260 (was 260), AvailableMemoryMB=3498 (was 3505), ConnectionCount=0 (was 0) 2013-10-04 14:30:38,825 INFO [pool-1-thread-1] hbase.ResourceChecker(147): before: master.TestSplitLogManager#testRecoveryRegionRemovedFromZK Thread=17, OpenFileDescriptor=120, MaxFileDescriptor=10240, SystemLoadAverage=964, ProcessCount=260, AvailableMemoryMB=3478, ConnectionCount=0 2013-10-04 14:30:38,849 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(341): System.getProperty("hadoop.log.dir") already set to: /Users/stack/checkouts/hbase/hbase-server/target/test-data/beb7a788-857e-44ec-a61e-5b7716b9d10e/hadoop-log-dir so I do NOT create it in target/test-data/186c9005-cc4f-48a9-b27a-adfd96b991e7 2013-10-04 14:30:38,849 WARN [pool-1-thread-1] hbase.HBaseTestingUtility(345): hadoop.log.dir property value differs in configuration and system: Configuration=/Users/stack/checkouts/hbase/hbase-server/../logs while System=/Users/stack/checkouts/hbase/hbase-server/target/test-data/beb7a788-857e-44ec-a61e-5b7716b9d10e/hadoop-log-dir Erasing configuration value by system value. 2013-10-04 14:30:38,849 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(341): System.getProperty("hadoop.tmp.dir") already set to: /Users/stack/checkouts/hbase/hbase-server/target/test-data/beb7a788-857e-44ec-a61e-5b7716b9d10e/hadoop-tmp-dir so I do NOT create it in target/test-data/186c9005-cc4f-48a9-b27a-adfd96b991e7 2013-10-04 14:30:38,849 WARN [pool-1-thread-1] hbase.HBaseTestingUtility(345): hadoop.tmp.dir property value differs in configuration and system: Configuration=/tmp/hadoop-stack while System=/Users/stack/checkouts/hbase/hbase-server/target/test-data/beb7a788-857e-44ec-a61e-5b7716b9d10e/hadoop-tmp-dir Erasing configuration value by system value. 2013-10-04 14:30:38,850 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(398): Created new mini-cluster data directory: /Users/stack/checkouts/hbase/hbase-server/target/test-data/186c9005-cc4f-48a9-b27a-adfd96b991e7/dfscluster_4d33ca55-4e2a-4802-bdb8-675ad962bcc1, deleteOnExit=true 2013-10-04 14:30:38,892 INFO [pool-1-thread-1] zookeeper.MiniZooKeeperCluster(200): Started MiniZK Cluster and connect 1 ZK server on client port: 60524 2013-10-04 14:30:38,904 INFO [pool-1-thread-1] zookeeper.RecoverableZooKeeper(120): Process identifier=split-log-manager-tests1a4d2415-27b6-4b3c-9a3b-0902c6cc6f9a connecting to ZooKeeper ensemble=localhost:60524 2013-10-04 14:30:38,907 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-tests1a4d2415-27b6-4b3c-9a3b-0902c6cc6f9a Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2013-10-04 14:30:38,914 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(387): split-log-manager-tests1a4d2415-27b6-4b3c-9a3b-0902c6cc6f9a-0x141856127860000 connected 2013-10-04 14:30:38,935 DEBUG [pool-1-thread-1] master.TestSplitLogManager(119): /hbase created 2013-10-04 14:30:38,960 DEBUG [pool-1-thread-1] master.TestSplitLogManager(122): /hbase/splitWAL created 2013-10-04 14:30:38,965 INFO [Thread-14] master.TestSplitLogManager(546): testRecoveryRegionRemovedFromZK 2013-10-04 14:30:38,983 INFO [Thread-14] master.SplitLogManager(227): Timeout=6000, unassigned timeout=12000, distributedLogReplay=false 2013-10-04 14:30:38,986 INFO [Thread-14] master.SplitLogManager(1108): Found 0 orphan tasks and 0 rescan nodes 2013-10-04 14:30:38,998 INFO [dummy-master,1,1.splitLogManagerTimeoutMonitor] hbase.Chore(93): dummy-master,1,1.splitLogManagerTimeoutMonitor exiting 2013-10-04 14:30:39,002 INFO [pool-1-thread-1] zookeeper.MiniZooKeeperCluster(249): Shutdown MiniZK cluster with all ZK servers 2013-10-04 14:30:39,101 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-tests1a4d2415-27b6-4b3c-9a3b-0902c6cc6f9a-0x141856127860000 Received ZooKeeper Event, type=None, state=Disconnected, path=null 2013-10-04 14:30:39,102 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(392): split-log-manager-tests1a4d2415-27b6-4b3c-9a3b-0902c6cc6f9a-0x141856127860000 Received Disconnected from ZooKeeper, ignoring 2013-10-04 14:30:39,172 INFO [pool-1-thread-1] hbase.ResourceChecker(171): after: master.TestSplitLogManager#testRecoveryRegionRemovedFromZK Thread=20 (was 17) Potentially hanging thread: pool-1-thread-1-SendThread(localhost:60524) java.lang.Thread.sleep(Native Method) org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:219) org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1157) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1097) - Thread LEAK? -, OpenFileDescriptor=126 (was 120) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=10240 (was 10240), SystemLoadAverage=964 (was 964), ProcessCount=260 (was 260), AvailableMemoryMB=3484 (was 3478) - AvailableMemoryMB LEAK? -, ConnectionCount=0 (was 0) 2013-10-04 14:30:39,327 INFO [pool-1-thread-1] hbase.ResourceChecker(147): before: master.TestSplitLogManager#testWorkerCrash Thread=20, OpenFileDescriptor=128, MaxFileDescriptor=10240, SystemLoadAverage=964, ProcessCount=260, AvailableMemoryMB=3473, ConnectionCount=0 2013-10-04 14:30:39,360 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(341): System.getProperty("hadoop.log.dir") already set to: /Users/stack/checkouts/hbase/hbase-server/target/test-data/beb7a788-857e-44ec-a61e-5b7716b9d10e/hadoop-log-dir so I do NOT create it in target/test-data/c015b88c-ac17-486d-9c41-48fb749fa886 2013-10-04 14:30:39,360 WARN [pool-1-thread-1] hbase.HBaseTestingUtility(345): hadoop.log.dir property value differs in configuration and system: Configuration=/Users/stack/checkouts/hbase/hbase-server/../logs while System=/Users/stack/checkouts/hbase/hbase-server/target/test-data/beb7a788-857e-44ec-a61e-5b7716b9d10e/hadoop-log-dir Erasing configuration value by system value. 2013-10-04 14:30:39,361 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(341): System.getProperty("hadoop.tmp.dir") already set to: /Users/stack/checkouts/hbase/hbase-server/target/test-data/beb7a788-857e-44ec-a61e-5b7716b9d10e/hadoop-tmp-dir so I do NOT create it in target/test-data/c015b88c-ac17-486d-9c41-48fb749fa886 2013-10-04 14:30:39,361 WARN [pool-1-thread-1] hbase.HBaseTestingUtility(345): hadoop.tmp.dir property value differs in configuration and system: Configuration=/tmp/hadoop-stack while System=/Users/stack/checkouts/hbase/hbase-server/target/test-data/beb7a788-857e-44ec-a61e-5b7716b9d10e/hadoop-tmp-dir Erasing configuration value by system value. 2013-10-04 14:30:39,361 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(398): Created new mini-cluster data directory: /Users/stack/checkouts/hbase/hbase-server/target/test-data/c015b88c-ac17-486d-9c41-48fb749fa886/dfscluster_46100af2-1105-47ec-bc00-69662ac523a6, deleteOnExit=true 2013-10-04 14:30:39,387 INFO [pool-1-thread-1] zookeeper.MiniZooKeeperCluster(200): Started MiniZK Cluster and connect 1 ZK server on client port: 57091 2013-10-04 14:30:39,390 INFO [pool-1-thread-1] zookeeper.RecoverableZooKeeper(120): Process identifier=split-log-manager-testsa962ffdc-5477-498e-b649-ad18aa7682cd connecting to ZooKeeper ensemble=localhost:57091 2013-10-04 14:30:39,395 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsa962ffdc-5477-498e-b649-ad18aa7682cd Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2013-10-04 14:30:39,399 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(387): split-log-manager-testsa962ffdc-5477-498e-b649-ad18aa7682cd-0x141856129860000 connected 2013-10-04 14:30:39,406 DEBUG [pool-1-thread-1] master.TestSplitLogManager(119): /hbase created 2013-10-04 14:30:39,412 DEBUG [pool-1-thread-1] master.TestSplitLogManager(122): /hbase/splitWAL created 2013-10-04 14:30:39,413 INFO [pool-1-thread-1] master.SplitLogManager(227): Timeout=6000, unassigned timeout=12000, distributedLogReplay=false 2013-10-04 14:30:39,422 INFO [pool-1-thread-1] master.SplitLogManager(1108): Found 0 orphan tasks and 0 rescan nodes 2013-10-04 14:30:39,424 DEBUG [pool-1-thread-1] zookeeper.ZKUtil(430): split-log-manager-testsa962ffdc-5477-498e-b649-ad18aa7682cd-0x141856129860000 Set watcher on znode that does not yet exist, /hbase/splitWAL/foo%2F1 2013-10-04 14:30:39,425 DEBUG [pool-1-thread-1] master.TestSplitLogManager(186): waiting for task node creation 2013-10-04 14:30:39,429 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsa962ffdc-5477-498e-b649-ad18aa7682cd-0x141856129860000 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/splitWAL/foo%2F1 2013-10-04 14:30:39,429 DEBUG [pool-1-thread-1-EventThread] regionserver.TestMasterAddressManager$NodeCreationListener(107): nodeCreated(/hbase/splitWAL/foo%2F1) 2013-10-04 14:30:39,429 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(711): put up splitlog task at znode /hbase/splitWAL/foo%2F1 2013-10-04 14:30:39,430 DEBUG [pool-1-thread-1] master.TestSplitLogManager(188): task created 2013-10-04 14:30:39,431 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(753): task not yet acquired /hbase/splitWAL/foo%2F1 ver = 0 2013-10-04 14:30:39,433 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsa962ffdc-5477-498e-b649-ad18aa7682cd-0x141856129860000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/foo%2F1 2013-10-04 14:30:39,436 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(826): task /hbase/splitWAL/foo%2F1 acquired by worker1,1,1 2013-10-04 14:30:39,521 INFO [dummy-master,1,1.splitLogManagerTimeoutMonitor] master.SplitLogManager(875): resubmitting task /hbase/splitWAL/foo%2F1 2013-10-04 14:30:39,525 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsa962ffdc-5477-498e-b649-ad18aa7682cd-0x141856129860000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/foo%2F1 2013-10-04 14:30:39,526 INFO [dummy-master,1,1.splitLogManagerTimeoutMonitor] master.SplitLogManager$TimeoutMonitor(1408): total tasks = 1 unassigned = 0 tasks={/hbase/splitWAL/foo%2F1=last_update = -1 last_version = 1 cur_worker_name = null status = in_progress incarnation = 1 resubmits = 1 batch = installed = 1 done = 0 error = 0} 2013-10-04 14:30:39,526 INFO [dummy-master,1,1.splitLogManagerTimeoutMonitor] master.SplitLogManager$TimeoutMonitor(1412): resubmitted 1 out of 1 tasks 2013-10-04 14:30:39,527 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(753): task not yet acquired /hbase/splitWAL/foo%2F1 ver = 2 2013-10-04 14:30:39,531 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(761): task /hbase/splitWAL/RESCAN0000000001 entered state: DONE dummy-master,1,1 2013-10-04 14:30:39,533 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsa962ffdc-5477-498e-b649-ad18aa7682cd-0x141856129860000 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/splitWAL/RESCAN0000000001 2013-10-04 14:30:39,533 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager$DeleteAsyncCallback(1605): deleted /hbase/splitWAL/RESCAN0000000001 2013-10-04 14:30:39,533 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(943): deleted task without in memory state /hbase/splitWAL/RESCAN0000000001 2013-10-04 14:30:40,739 INFO [dummy-master,1,1.splitLogManagerTimeoutMonitor] hbase.Chore(93): dummy-master,1,1.splitLogManagerTimeoutMonitor exiting 2013-10-04 14:30:40,741 INFO [pool-1-thread-1] zookeeper.MiniZooKeeperCluster(249): Shutdown MiniZK cluster with all ZK servers 2013-10-04 14:30:40,841 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsa962ffdc-5477-498e-b649-ad18aa7682cd-0x141856129860000 Received ZooKeeper Event, type=None, state=Disconnected, path=null 2013-10-04 14:30:40,841 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(392): split-log-manager-testsa962ffdc-5477-498e-b649-ad18aa7682cd-0x141856129860000 Received Disconnected from ZooKeeper, ignoring 2013-10-04 14:30:40,868 INFO [pool-1-thread-1] hbase.ResourceChecker(171): after: master.TestSplitLogManager#testWorkerCrash Thread=21 (was 20) Potentially hanging thread: pool-1-thread-1-SendThread(localhost:57091) java.lang.Thread.sleep(Native Method) org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:219) org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1157) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1097) - Thread LEAK? -, OpenFileDescriptor=124 (was 128), MaxFileDescriptor=10240 (was 10240), SystemLoadAverage=964 (was 964), ProcessCount=260 (was 260), AvailableMemoryMB=3456 (was 3473), ConnectionCount=0 (was 0) 2013-10-04 14:30:41,011 INFO [pool-1-thread-1] hbase.ResourceChecker(147): before: master.TestSplitLogManager#testTaskDone Thread=21, OpenFileDescriptor=126, MaxFileDescriptor=10240, SystemLoadAverage=964, ProcessCount=260, AvailableMemoryMB=3448, ConnectionCount=0 2013-10-04 14:30:41,034 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(341): System.getProperty("hadoop.log.dir") already set to: /Users/stack/checkouts/hbase/hbase-server/target/test-data/beb7a788-857e-44ec-a61e-5b7716b9d10e/hadoop-log-dir so I do NOT create it in target/test-data/36b580f4-549d-41b4-8415-e89d7119f732 2013-10-04 14:30:41,035 WARN [pool-1-thread-1] hbase.HBaseTestingUtility(345): hadoop.log.dir property value differs in configuration and system: Configuration=/Users/stack/checkouts/hbase/hbase-server/../logs while System=/Users/stack/checkouts/hbase/hbase-server/target/test-data/beb7a788-857e-44ec-a61e-5b7716b9d10e/hadoop-log-dir Erasing configuration value by system value. 2013-10-04 14:30:41,035 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(341): System.getProperty("hadoop.tmp.dir") already set to: /Users/stack/checkouts/hbase/hbase-server/target/test-data/beb7a788-857e-44ec-a61e-5b7716b9d10e/hadoop-tmp-dir so I do NOT create it in target/test-data/36b580f4-549d-41b4-8415-e89d7119f732 2013-10-04 14:30:41,036 WARN [pool-1-thread-1] hbase.HBaseTestingUtility(345): hadoop.tmp.dir property value differs in configuration and system: Configuration=/tmp/hadoop-stack while System=/Users/stack/checkouts/hbase/hbase-server/target/test-data/beb7a788-857e-44ec-a61e-5b7716b9d10e/hadoop-tmp-dir Erasing configuration value by system value. 2013-10-04 14:30:41,036 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(398): Created new mini-cluster data directory: /Users/stack/checkouts/hbase/hbase-server/target/test-data/36b580f4-549d-41b4-8415-e89d7119f732/dfscluster_55f77e1a-942f-44b0-889e-a8f5325339f8, deleteOnExit=true 2013-10-04 14:30:41,049 INFO [pool-1-thread-1] zookeeper.MiniZooKeeperCluster(200): Started MiniZK Cluster and connect 1 ZK server on client port: 58045 2013-10-04 14:30:41,051 INFO [pool-1-thread-1] zookeeper.RecoverableZooKeeper(120): Process identifier=split-log-manager-tests6dd4eaf2-99d5-4581-9932-49c25b6a0ac9 connecting to ZooKeeper ensemble=localhost:58045 2013-10-04 14:30:41,056 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-tests6dd4eaf2-99d5-4581-9932-49c25b6a0ac9 Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2013-10-04 14:30:41,059 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(387): split-log-manager-tests6dd4eaf2-99d5-4581-9932-49c25b6a0ac9-0x141856130100000 connected 2013-10-04 14:30:41,067 DEBUG [pool-1-thread-1] master.TestSplitLogManager(119): /hbase created 2013-10-04 14:30:41,074 DEBUG [pool-1-thread-1] master.TestSplitLogManager(122): /hbase/splitWAL created 2013-10-04 14:30:41,075 INFO [pool-1-thread-1] master.TestSplitLogManager(328): TestTaskDone - cleanup task node once in DONE state 2013-10-04 14:30:41,075 INFO [pool-1-thread-1] master.SplitLogManager(227): Timeout=6000, unassigned timeout=12000, distributedLogReplay=false 2013-10-04 14:30:41,078 INFO [pool-1-thread-1] master.SplitLogManager(1108): Found 0 orphan tasks and 0 rescan nodes 2013-10-04 14:30:41,080 DEBUG [pool-1-thread-1] zookeeper.ZKUtil(430): split-log-manager-tests6dd4eaf2-99d5-4581-9932-49c25b6a0ac9-0x141856130100000 Set watcher on znode that does not yet exist, /hbase/splitWAL/foo%2F1 2013-10-04 14:30:41,081 DEBUG [pool-1-thread-1] master.TestSplitLogManager(186): waiting for task node creation 2013-10-04 14:30:41,083 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-tests6dd4eaf2-99d5-4581-9932-49c25b6a0ac9-0x141856130100000 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/splitWAL/foo%2F1 2013-10-04 14:30:41,083 DEBUG [pool-1-thread-1-EventThread] regionserver.TestMasterAddressManager$NodeCreationListener(107): nodeCreated(/hbase/splitWAL/foo%2F1) 2013-10-04 14:30:41,083 DEBUG [pool-1-thread-1] master.TestSplitLogManager(188): task created 2013-10-04 14:30:41,083 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(711): put up splitlog task at znode /hbase/splitWAL/foo%2F1 2013-10-04 14:30:41,085 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(753): task not yet acquired /hbase/splitWAL/foo%2F1 ver = 0 2013-10-04 14:30:41,086 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-tests6dd4eaf2-99d5-4581-9932-49c25b6a0ac9-0x141856130100000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/foo%2F1 2013-10-04 14:30:41,088 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(761): task /hbase/splitWAL/foo%2F1 entered state: DONE worker1,1,1 2013-10-04 14:30:41,088 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(674): Done splitting /hbase/splitWAL/foo%2F1 2013-10-04 14:30:41,088 INFO [pool-1-thread-1] hbase.Waiter(174): Waiting up to [3,200] milli-secs(wait.for.ratio=[1]) 2013-10-04 14:30:41,090 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-tests6dd4eaf2-99d5-4581-9932-49c25b6a0ac9-0x141856130100000 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/splitWAL/foo%2F1 2013-10-04 14:30:41,090 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager$DeleteAsyncCallback(1605): deleted /hbase/splitWAL/foo%2F1 2013-10-04 14:30:41,101 INFO [dummy-master,1,1.splitLogManagerTimeoutMonitor] hbase.Chore(93): dummy-master,1,1.splitLogManagerTimeoutMonitor exiting 2013-10-04 14:30:41,106 INFO [pool-1-thread-1] zookeeper.MiniZooKeeperCluster(249): Shutdown MiniZK cluster with all ZK servers 2013-10-04 14:30:41,203 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-tests6dd4eaf2-99d5-4581-9932-49c25b6a0ac9-0x141856130100000 Received ZooKeeper Event, type=None, state=Disconnected, path=null 2013-10-04 14:30:41,203 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(392): split-log-manager-tests6dd4eaf2-99d5-4581-9932-49c25b6a0ac9-0x141856130100000 Received Disconnected from ZooKeeper, ignoring 2013-10-04 14:30:41,236 INFO [pool-1-thread-1] hbase.ResourceChecker(171): after: master.TestSplitLogManager#testTaskDone Thread=24 (was 21) Potentially hanging thread: pool-1-thread-1-SendThread(localhost:58045) java.lang.Thread.sleep(Native Method) org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:219) org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1157) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1097) - Thread LEAK? -, OpenFileDescriptor=132 (was 126) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=10240 (was 10240), SystemLoadAverage=964 (was 964), ProcessCount=260 (was 260), AvailableMemoryMB=3449 (was 3448) - AvailableMemoryMB LEAK? -, ConnectionCount=0 (was 0) 2013-10-04 14:30:41,359 INFO [pool-1-thread-1] hbase.ResourceChecker(147): before: master.TestSplitLogManager#testEmptyLogDir Thread=24, OpenFileDescriptor=134, MaxFileDescriptor=10240, SystemLoadAverage=964, ProcessCount=260, AvailableMemoryMB=3448, ConnectionCount=0 2013-10-04 14:30:41,382 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(341): System.getProperty("hadoop.log.dir") already set to: /Users/stack/checkouts/hbase/hbase-server/target/test-data/beb7a788-857e-44ec-a61e-5b7716b9d10e/hadoop-log-dir so I do NOT create it in target/test-data/2a8d6811-8c07-4354-a175-c2735dd7cc3d 2013-10-04 14:30:41,383 WARN [pool-1-thread-1] hbase.HBaseTestingUtility(345): hadoop.log.dir property value differs in configuration and system: Configuration=/Users/stack/checkouts/hbase/hbase-server/../logs while System=/Users/stack/checkouts/hbase/hbase-server/target/test-data/beb7a788-857e-44ec-a61e-5b7716b9d10e/hadoop-log-dir Erasing configuration value by system value. 2013-10-04 14:30:41,383 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(341): System.getProperty("hadoop.tmp.dir") already set to: /Users/stack/checkouts/hbase/hbase-server/target/test-data/beb7a788-857e-44ec-a61e-5b7716b9d10e/hadoop-tmp-dir so I do NOT create it in target/test-data/2a8d6811-8c07-4354-a175-c2735dd7cc3d 2013-10-04 14:30:41,383 WARN [pool-1-thread-1] hbase.HBaseTestingUtility(345): hadoop.tmp.dir property value differs in configuration and system: Configuration=/tmp/hadoop-stack while System=/Users/stack/checkouts/hbase/hbase-server/target/test-data/beb7a788-857e-44ec-a61e-5b7716b9d10e/hadoop-tmp-dir Erasing configuration value by system value. 2013-10-04 14:30:41,384 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(398): Created new mini-cluster data directory: /Users/stack/checkouts/hbase/hbase-server/target/test-data/2a8d6811-8c07-4354-a175-c2735dd7cc3d/dfscluster_52525bbe-48ff-4088-8025-105de87697f1, deleteOnExit=true 2013-10-04 14:30:41,392 INFO [pool-1-thread-1] zookeeper.MiniZooKeeperCluster(200): Started MiniZK Cluster and connect 1 ZK server on client port: 56653 2013-10-04 14:30:41,394 INFO [pool-1-thread-1] zookeeper.RecoverableZooKeeper(120): Process identifier=split-log-manager-tests63460c82-34f7-4ced-a035-debf340dbb79 connecting to ZooKeeper ensemble=localhost:56653 2013-10-04 14:30:41,400 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-tests63460c82-34f7-4ced-a035-debf340dbb79 Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2013-10-04 14:30:41,402 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(387): split-log-manager-tests63460c82-34f7-4ced-a035-debf340dbb79-0x1418561316b0000 connected 2013-10-04 14:30:41,410 DEBUG [pool-1-thread-1] master.TestSplitLogManager(119): /hbase created 2013-10-04 14:30:41,416 DEBUG [pool-1-thread-1] master.TestSplitLogManager(122): /hbase/splitWAL created 2013-10-04 14:30:41,418 INFO [pool-1-thread-1] master.TestSplitLogManager(484): testEmptyLogDir 2013-10-04 14:30:41,418 INFO [pool-1-thread-1] master.SplitLogManager(227): Timeout=6000, unassigned timeout=12000, distributedLogReplay=false 2013-10-04 14:30:41,420 INFO [pool-1-thread-1] master.SplitLogManager(1108): Found 0 orphan tasks and 0 rescan nodes 2013-10-04 14:30:41,421 DEBUG [pool-1-thread-1] fs.HFileSystem(213): The file system is not a DistributedFileSystem. Skipping on block location reordering 2013-10-04 14:30:41,422 WARN [pool-1-thread-1] wal.HLogUtil(189): Invalid log file path=file:/Users/stack/checkouts/hbase/hbase-server/bff55468-99cd-4314-bf99-e30671bff3d1 java.lang.IllegalArgumentException: Not a host:port pair: hbase-server at org.apache.hadoop.hbase.util.Addressing.parseHostname(Addressing.java:62) at org.apache.hadoop.hbase.ServerName.(ServerName.java:125) at org.apache.hadoop.hbase.ServerName.parseServerName(ServerName.java:318) at org.apache.hadoop.hbase.regionserver.wal.HLogUtil.getServerNameFromHLogDirectoryName(HLogUtil.java:186) at org.apache.hadoop.hbase.master.SplitLogManager.splitLogDistributed(SplitLogManager.java:298) at org.apache.hadoop.hbase.master.SplitLogManager.splitLogDistributed(SplitLogManager.java:278) at org.apache.hadoop.hbase.master.TestSplitLogManager.testEmptyLogDir(TestSplitLogManager.java:490) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at org.junit.runners.ParentRunner.run(ParentRunner.java:309) at org.junit.runners.Suite.runChild(Suite.java:127) at org.junit.runners.Suite.runChild(Suite.java:26) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:680) 2013-10-04 14:30:41,424 INFO [pool-1-thread-1] master.SplitLogManager(257): file:/Users/stack/checkouts/hbase/hbase-server/bff55468-99cd-4314-bf99-e30671bff3d1 is empty dir, no logs to split 2013-10-04 14:30:41,424 DEBUG [pool-1-thread-1] master.SplitLogManager(327): Scheduling batch of logs to split 2013-10-04 14:30:41,425 INFO [pool-1-thread-1] master.SplitLogManager(329): started splitting 0 logs in [file:/Users/stack/checkouts/hbase/hbase-server/bff55468-99cd-4314-bf99-e30671bff3d1] 2013-10-04 14:30:41,425 INFO [pool-1-thread-1] master.SplitLogManager(386): finished splitting (more than or equal to) 0 bytes in 0 log files in [file:/Users/stack/checkouts/hbase/hbase-server/bff55468-99cd-4314-bf99-e30671bff3d1] in 0ms 2013-10-04 14:30:41,426 INFO [dummy-master,1,1.splitLogManagerTimeoutMonitor] hbase.Chore(93): dummy-master,1,1.splitLogManagerTimeoutMonitor exiting 2013-10-04 14:30:41,432 INFO [pool-1-thread-1] zookeeper.MiniZooKeeperCluster(249): Shutdown MiniZK cluster with all ZK servers 2013-10-04 14:30:41,529 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-tests63460c82-34f7-4ced-a035-debf340dbb79-0x1418561316b0000 Received ZooKeeper Event, type=None, state=Disconnected, path=null 2013-10-04 14:30:41,530 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(392): split-log-manager-tests63460c82-34f7-4ced-a035-debf340dbb79-0x1418561316b0000 Received Disconnected from ZooKeeper, ignoring 2013-10-04 14:30:41,558 INFO [pool-1-thread-1] hbase.ResourceChecker(171): after: master.TestSplitLogManager#testEmptyLogDir Thread=27 (was 24) Potentially hanging thread: pool-1-thread-1-SendThread(localhost:56653) java.lang.Thread.sleep(Native Method) org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:219) org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1157) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1097) - Thread LEAK? -, OpenFileDescriptor=140 (was 134) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=10240 (was 10240), SystemLoadAverage=964 (was 964), ProcessCount=260 (was 260), AvailableMemoryMB=3447 (was 3448), ConnectionCount=0 (was 0) 2013-10-04 14:30:41,685 INFO [pool-1-thread-1] hbase.ResourceChecker(147): before: master.TestSplitLogManager#testOrphanTaskAcquisition Thread=27, OpenFileDescriptor=142, MaxFileDescriptor=10240, SystemLoadAverage=964, ProcessCount=260, AvailableMemoryMB=3445, ConnectionCount=0 2013-10-04 14:30:41,705 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(341): System.getProperty("hadoop.log.dir") already set to: /Users/stack/checkouts/hbase/hbase-server/target/test-data/beb7a788-857e-44ec-a61e-5b7716b9d10e/hadoop-log-dir so I do NOT create it in target/test-data/d6ccbfd5-1fb1-4c82-b79b-b4193d2a7eb5 2013-10-04 14:30:41,705 WARN [pool-1-thread-1] hbase.HBaseTestingUtility(345): hadoop.log.dir property value differs in configuration and system: Configuration=/Users/stack/checkouts/hbase/hbase-server/../logs while System=/Users/stack/checkouts/hbase/hbase-server/target/test-data/beb7a788-857e-44ec-a61e-5b7716b9d10e/hadoop-log-dir Erasing configuration value by system value. 2013-10-04 14:30:41,705 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(341): System.getProperty("hadoop.tmp.dir") already set to: /Users/stack/checkouts/hbase/hbase-server/target/test-data/beb7a788-857e-44ec-a61e-5b7716b9d10e/hadoop-tmp-dir so I do NOT create it in target/test-data/d6ccbfd5-1fb1-4c82-b79b-b4193d2a7eb5 2013-10-04 14:30:41,705 WARN [pool-1-thread-1] hbase.HBaseTestingUtility(345): hadoop.tmp.dir property value differs in configuration and system: Configuration=/tmp/hadoop-stack while System=/Users/stack/checkouts/hbase/hbase-server/target/test-data/beb7a788-857e-44ec-a61e-5b7716b9d10e/hadoop-tmp-dir Erasing configuration value by system value. 2013-10-04 14:30:41,706 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(398): Created new mini-cluster data directory: /Users/stack/checkouts/hbase/hbase-server/target/test-data/d6ccbfd5-1fb1-4c82-b79b-b4193d2a7eb5/dfscluster_54f1a450-6d1f-4f01-84c0-08dba2c2f896, deleteOnExit=true 2013-10-04 14:30:41,726 INFO [pool-1-thread-1] zookeeper.MiniZooKeeperCluster(200): Started MiniZK Cluster and connect 1 ZK server on client port: 59024 2013-10-04 14:30:41,728 INFO [pool-1-thread-1] zookeeper.RecoverableZooKeeper(120): Process identifier=split-log-manager-tests79aa252c-2117-4cb4-8f2a-021862c5d61d connecting to ZooKeeper ensemble=localhost:59024 2013-10-04 14:30:41,732 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-tests79aa252c-2117-4cb4-8f2a-021862c5d61d Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2013-10-04 14:30:41,734 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(387): split-log-manager-tests79aa252c-2117-4cb4-8f2a-021862c5d61d-0x141856132b80000 connected 2013-10-04 14:30:41,739 DEBUG [pool-1-thread-1] master.TestSplitLogManager(119): /hbase created 2013-10-04 14:30:41,746 DEBUG [pool-1-thread-1] master.TestSplitLogManager(122): /hbase/splitWAL created 2013-10-04 14:30:41,747 INFO [pool-1-thread-1] master.TestSplitLogManager(213): TestOrphanTaskAcquisition 2013-10-04 14:30:41,750 INFO [pool-1-thread-1] master.SplitLogManager(227): Timeout=6000, unassigned timeout=12000, distributedLogReplay=false 2013-10-04 14:30:41,753 INFO [pool-1-thread-1] master.SplitLogManager(1104): found orphan task orphan%2Ftest%2Fslash 2013-10-04 14:30:41,754 INFO [pool-1-thread-1] master.SplitLogManager(1108): Found 1 orphan tasks and 0 rescan nodes 2013-10-04 14:30:41,754 INFO [pool-1-thread-1] hbase.Waiter(174): Waiting up to [3,200] milli-secs(wait.for.ratio=[1]) 2013-10-04 14:30:41,755 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(1058): creating orphan task /hbase/splitWAL/orphan%2Ftest%2Fslash 2013-10-04 14:30:41,755 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(826): task /hbase/splitWAL/orphan%2Ftest%2Fslash acquired by dummy-master,1,1 2013-10-04 14:30:41,765 INFO [pool-1-thread-1] hbase.Waiter(174): Waiting up to [3,200] milli-secs(wait.for.ratio=[1]) 2013-10-04 14:30:41,766 INFO [pool-1-thread-1] master.TestSplitLogManager(229): waiting for manager to resubmit the orphan task 2013-10-04 14:30:41,766 INFO [pool-1-thread-1] hbase.Waiter(174): Waiting up to [9,600] milli-secs(wait.for.ratio=[1]) 2013-10-04 14:30:41,854 INFO [dummy-master,1,1.splitLogManagerTimeoutMonitor] master.SplitLogManager$TimeoutMonitor(1408): total tasks = 1 unassigned = 0 tasks={/hbase/splitWAL/orphan%2Ftest%2Fslash=last_update = 1380922241755 last_version = 0 cur_worker_name = dummy-master,1,1 status = in_progress incarnation = 0 resubmits = 0 batch = null} 2013-10-04 14:30:46,892 INFO [dummy-master,1,1.splitLogManagerTimeoutMonitor] master.SplitLogManager$TimeoutMonitor(1408): total tasks = 1 unassigned = 0 tasks={/hbase/splitWAL/orphan%2Ftest%2Fslash=last_update = 1380922241755 last_version = 0 cur_worker_name = dummy-master,1,1 status = in_progress incarnation = 0 resubmits = 0 batch = null} 2013-10-04 14:30:47,799 INFO [dummy-master,1,1.splitLogManagerTimeoutMonitor] master.SplitLogManager(875): resubmitting task /hbase/splitWAL/orphan%2Ftest%2Fslash 2013-10-04 14:30:47,801 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-tests79aa252c-2117-4cb4-8f2a-021862c5d61d-0x141856132b80000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2Ftest%2Fslash 2013-10-04 14:30:47,801 INFO [dummy-master,1,1.splitLogManagerTimeoutMonitor] master.SplitLogManager$TimeoutMonitor(1412): resubmitted 1 out of 1 tasks 2013-10-04 14:30:47,802 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(753): task not yet acquired /hbase/splitWAL/orphan%2Ftest%2Fslash ver = 1 2013-10-04 14:30:47,804 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(761): task /hbase/splitWAL/RESCAN0000000001 entered state: DONE dummy-master,1,1 2013-10-04 14:30:47,805 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-tests79aa252c-2117-4cb4-8f2a-021862c5d61d-0x141856132b80000 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/splitWAL/RESCAN0000000001 2013-10-04 14:30:47,805 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager$DeleteAsyncCallback(1605): deleted /hbase/splitWAL/RESCAN0000000001 2013-10-04 14:30:47,805 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(943): deleted task without in memory state /hbase/splitWAL/RESCAN0000000001 2013-10-04 14:30:47,812 INFO [pool-1-thread-1] hbase.Waiter(174): Waiting up to [9,600] milli-secs(wait.for.ratio=[1]) 2013-10-04 14:30:47,812 INFO [dummy-master,1,1.splitLogManagerTimeoutMonitor] hbase.Chore(93): dummy-master,1,1.splitLogManagerTimeoutMonitor exiting 2013-10-04 14:30:47,814 INFO [pool-1-thread-1] zookeeper.MiniZooKeeperCluster(249): Shutdown MiniZK cluster with all ZK servers 2013-10-04 14:30:47,881 INFO [pool-1-thread-1] hbase.ResourceChecker(171): after: master.TestSplitLogManager#testOrphanTaskAcquisition Thread=27 (was 27), OpenFileDescriptor=148 (was 142) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=10240 (was 10240), SystemLoadAverage=943 (was 964), ProcessCount=260 (was 260), AvailableMemoryMB=3391 (was 3445), ConnectionCount=0 (was 0) 2013-10-04 14:30:47,914 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-tests79aa252c-2117-4cb4-8f2a-021862c5d61d-0x141856132b80000 Received ZooKeeper Event, type=None, state=Disconnected, path=null 2013-10-04 14:30:47,915 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(392): split-log-manager-tests79aa252c-2117-4cb4-8f2a-021862c5d61d-0x141856132b80000 Received Disconnected from ZooKeeper, ignoring 2013-10-04 14:30:47,950 INFO [pool-1-thread-1] hbase.ResourceChecker(147): before: master.TestSplitLogManager#testUnassignedTimeout Thread=27, OpenFileDescriptor=150, MaxFileDescriptor=10240, SystemLoadAverage=943, ProcessCount=260, AvailableMemoryMB=3392, ConnectionCount=0 2013-10-04 14:30:47,962 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(341): System.getProperty("hadoop.log.dir") already set to: /Users/stack/checkouts/hbase/hbase-server/target/test-data/beb7a788-857e-44ec-a61e-5b7716b9d10e/hadoop-log-dir so I do NOT create it in target/test-data/5185e51c-7f22-4c0d-ac34-c883e39a04cf 2013-10-04 14:30:47,962 WARN [pool-1-thread-1] hbase.HBaseTestingUtility(345): hadoop.log.dir property value differs in configuration and system: Configuration=/Users/stack/checkouts/hbase/hbase-server/../logs while System=/Users/stack/checkouts/hbase/hbase-server/target/test-data/beb7a788-857e-44ec-a61e-5b7716b9d10e/hadoop-log-dir Erasing configuration value by system value. 2013-10-04 14:30:47,962 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(341): System.getProperty("hadoop.tmp.dir") already set to: /Users/stack/checkouts/hbase/hbase-server/target/test-data/beb7a788-857e-44ec-a61e-5b7716b9d10e/hadoop-tmp-dir so I do NOT create it in target/test-data/5185e51c-7f22-4c0d-ac34-c883e39a04cf 2013-10-04 14:30:47,962 WARN [pool-1-thread-1] hbase.HBaseTestingUtility(345): hadoop.tmp.dir property value differs in configuration and system: Configuration=/tmp/hadoop-stack while System=/Users/stack/checkouts/hbase/hbase-server/target/test-data/beb7a788-857e-44ec-a61e-5b7716b9d10e/hadoop-tmp-dir Erasing configuration value by system value. 2013-10-04 14:30:47,962 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(398): Created new mini-cluster data directory: /Users/stack/checkouts/hbase/hbase-server/target/test-data/5185e51c-7f22-4c0d-ac34-c883e39a04cf/dfscluster_48ae16f1-cce2-4dcc-8a98-7637225fd816, deleteOnExit=true 2013-10-04 14:30:47,980 INFO [pool-1-thread-1] zookeeper.MiniZooKeeperCluster(200): Started MiniZK Cluster and connect 1 ZK server on client port: 49603 2013-10-04 14:30:47,982 INFO [pool-1-thread-1] zookeeper.RecoverableZooKeeper(120): Process identifier=split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375 connecting to ZooKeeper ensemble=localhost:49603 2013-10-04 14:30:47,986 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375 Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2013-10-04 14:30:47,988 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(387): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 connected 2013-10-04 14:30:47,990 DEBUG [pool-1-thread-1] master.TestSplitLogManager(119): /hbase created 2013-10-04 14:30:47,993 DEBUG [pool-1-thread-1] master.TestSplitLogManager(122): /hbase/splitWAL created 2013-10-04 14:30:47,994 INFO [pool-1-thread-1] master.TestSplitLogManager(396): TestUnassignedTimeout - iff all tasks are unassigned then resubmit 2013-10-04 14:30:47,996 INFO [pool-1-thread-1] master.SplitLogManager(227): Timeout=6000, unassigned timeout=12000, distributedLogReplay=false 2013-10-04 14:30:47,997 INFO [pool-1-thread-1] master.SplitLogManager(1104): found orphan task orphan%2F1 2013-10-04 14:30:47,997 INFO [pool-1-thread-1] master.SplitLogManager(1108): Found 1 orphan tasks and 0 rescan nodes 2013-10-04 14:30:47,998 INFO [pool-1-thread-1] hbase.Waiter(174): Waiting up to [3,200] milli-secs(wait.for.ratio=[1]) 2013-10-04 14:30:47,998 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(1058): creating orphan task /hbase/splitWAL/orphan%2F1 2013-10-04 14:30:47,998 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(826): task /hbase/splitWAL/orphan%2F1 acquired by worker1,1,1 2013-10-04 14:30:48,009 DEBUG [pool-1-thread-1] zookeeper.ZKUtil(430): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Set watcher on znode that does not yet exist, /hbase/splitWAL/foo%2F1 2013-10-04 14:30:48,010 DEBUG [pool-1-thread-1] master.TestSplitLogManager(186): waiting for task node creation 2013-10-04 14:30:48,011 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/splitWAL/foo%2F1 2013-10-04 14:30:48,011 DEBUG [pool-1-thread-1-EventThread] regionserver.TestMasterAddressManager$NodeCreationListener(107): nodeCreated(/hbase/splitWAL/foo%2F1) 2013-10-04 14:30:48,011 DEBUG [pool-1-thread-1] master.TestSplitLogManager(188): task created 2013-10-04 14:30:48,011 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(711): put up splitlog task at znode /hbase/splitWAL/foo%2F1 2013-10-04 14:30:48,012 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(753): task not yet acquired /hbase/splitWAL/foo%2F1 ver = 0 2013-10-04 14:30:48,099 INFO [dummy-master,1,1.splitLogManagerTimeoutMonitor] master.SplitLogManager$TimeoutMonitor(1408): total tasks = 2 unassigned = 1 tasks={/hbase/splitWAL/orphan%2F1=last_update = 1380922247998 last_version = 0 cur_worker_name = worker1,1,1 status = in_progress incarnation = 0 resubmits = 0 batch = null, /hbase/splitWAL/foo%2F1=last_update = -1 last_version = -1 cur_worker_name = null status = in_progress incarnation = 0 resubmits = 0 batch = installed = 1 done = 0 error = 0} 2013-10-04 14:30:48,114 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:48,217 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:48,319 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:48,421 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:48,524 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:48,626 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:48,729 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:48,831 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:48,935 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:49,037 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:49,140 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:49,243 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:49,345 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:49,449 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:49,551 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:49,654 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:49,758 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:49,861 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:49,964 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:50,067 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:50,171 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:50,274 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:50,376 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:50,479 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:50,584 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:50,687 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:50,791 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:50,895 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:50,997 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:51,102 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:51,205 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:51,308 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:51,411 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:51,514 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:51,617 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:51,720 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:51,824 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:51,926 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:52,030 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:52,132 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:52,234 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:52,338 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:52,441 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:52,544 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:52,647 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:52,750 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:52,852 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:52,955 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:53,059 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:53,136 INFO [dummy-master,1,1.splitLogManagerTimeoutMonitor] master.SplitLogManager$TimeoutMonitor(1408): total tasks = 2 unassigned = 1 tasks={/hbase/splitWAL/orphan%2F1=last_update = 1380922253060 last_version = 49 cur_worker_name = worker1,1,1 status = in_progress incarnation = 0 resubmits = 0 batch = null, /hbase/splitWAL/foo%2F1=last_update = -1 last_version = -1 cur_worker_name = null status = in_progress incarnation = 0 resubmits = 0 batch = installed = 1 done = 0 error = 0} 2013-10-04 14:30:53,161 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:53,265 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:53,392 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:53,495 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:53,597 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:53,700 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:53,803 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:53,906 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:54,010 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:54,112 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:54,215 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:54,317 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:54,422 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:54,524 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:54,627 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:54,730 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:54,832 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:54,937 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:55,041 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:55,143 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:55,245 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:55,348 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:55,452 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:55,557 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:55,660 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:55,764 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:55,867 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:55,970 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:56,072 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:56,174 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:56,277 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:56,380 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:56,482 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:56,585 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:56,689 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:56,792 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:56,894 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:56,997 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:57,099 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:57,202 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:57,304 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:57,407 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:57,509 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:57,612 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:57,717 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:57,819 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:57,922 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:58,024 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:58,128 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:58,171 INFO [dummy-master,1,1.splitLogManagerTimeoutMonitor] master.SplitLogManager$TimeoutMonitor(1408): total tasks = 2 unassigned = 1 tasks={/hbase/splitWAL/orphan%2F1=last_update = 1380922258129 last_version = 98 cur_worker_name = worker1,1,1 status = in_progress incarnation = 0 resubmits = 0 batch = null, /hbase/splitWAL/foo%2F1=last_update = -1 last_version = -1 cur_worker_name = null status = in_progress incarnation = 0 resubmits = 0 batch = installed = 1 done = 0 error = 0} 2013-10-04 14:30:58,231 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:58,333 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:58,436 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:58,540 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:58,642 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:58,745 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:58,848 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:58,951 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:59,054 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:59,157 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:59,259 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:59,361 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:59,464 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:59,567 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:59,670 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:59,774 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:59,877 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:30:59,980 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:00,084 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:00,187 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:00,289 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:00,392 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:00,494 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:00,596 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:00,698 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:00,801 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:00,904 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:01,007 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:01,110 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:01,213 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:01,316 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:01,418 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:01,521 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:01,624 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:01,726 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:01,829 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:01,931 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:02,033 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:02,143 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:02,247 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:02,349 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:02,455 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:02,557 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:02,660 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:02,763 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:02,866 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:02,968 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:03,073 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:03,178 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:03,217 INFO [dummy-master,1,1.splitLogManagerTimeoutMonitor] master.SplitLogManager$TimeoutMonitor(1408): total tasks = 2 unassigned = 1 tasks={/hbase/splitWAL/orphan%2F1=last_update = 1380922263179 last_version = 147 cur_worker_name = worker1,1,1 status = in_progress incarnation = 0 resubmits = 0 batch = null, /hbase/splitWAL/foo%2F1=last_update = -1 last_version = -1 cur_worker_name = null status = in_progress incarnation = 0 resubmits = 0 batch = installed = 1 done = 0 error = 0} 2013-10-04 14:31:03,281 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:03,383 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:03,486 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:03,589 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:03,693 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:03,795 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:03,897 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:03,999 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:04,100 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:04,203 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:04,305 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:04,408 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:04,511 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:04,614 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:04,718 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:04,824 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:04,926 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:05,030 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:05,134 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:05,238 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:05,340 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:05,444 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:05,548 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:05,651 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:05,754 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:05,862 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:05,966 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:06,070 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:06,173 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:06,276 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:06,378 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:06,480 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:06,583 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:06,686 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:06,789 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:06,891 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:06,994 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:07,096 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:07,199 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:07,304 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:07,406 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:07,510 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:07,613 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:07,718 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:07,820 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:07,821 INFO [pool-1-thread-1] master.TestSplitLogManager(423): waiting for manager to resubmit the orphan task 2013-10-04 14:31:07,821 INFO [pool-1-thread-1] hbase.Waiter(174): Waiting up to [9,600] milli-secs(wait.for.ratio=[1]) 2013-10-04 14:31:08,250 INFO [dummy-master,1,1.splitLogManagerTimeoutMonitor] master.SplitLogManager$TimeoutMonitor(1408): total tasks = 2 unassigned = 1 tasks={/hbase/splitWAL/orphan%2F1=last_update = 1380922267822 last_version = 192 cur_worker_name = worker1,1,1 status = in_progress incarnation = 0 resubmits = 0 batch = null, /hbase/splitWAL/foo%2F1=last_update = -1 last_version = -1 cur_worker_name = null status = in_progress incarnation = 0 resubmits = 0 batch = installed = 1 done = 0 error = 0} 2013-10-04 14:31:13,297 INFO [dummy-master,1,1.splitLogManagerTimeoutMonitor] master.SplitLogManager$TimeoutMonitor(1408): total tasks = 2 unassigned = 1 tasks={/hbase/splitWAL/orphan%2F1=last_update = 1380922267822 last_version = 192 cur_worker_name = worker1,1,1 status = in_progress incarnation = 0 resubmits = 0 batch = null, /hbase/splitWAL/foo%2F1=last_update = -1 last_version = -1 cur_worker_name = null status = in_progress incarnation = 0 resubmits = 0 batch = installed = 1 done = 0 error = 0} 2013-10-04 14:31:13,903 INFO [dummy-master,1,1.splitLogManagerTimeoutMonitor] master.SplitLogManager(875): resubmitting task /hbase/splitWAL/orphan%2F1 2013-10-04 14:31:13,904 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/orphan%2F1 2013-10-04 14:31:13,905 INFO [dummy-master,1,1.splitLogManagerTimeoutMonitor] master.SplitLogManager$TimeoutMonitor(1412): resubmitted 1 out of 2 tasks 2013-10-04 14:31:13,906 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(753): task not yet acquired /hbase/splitWAL/orphan%2F1 ver = 193 2013-10-04 14:31:13,907 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(761): task /hbase/splitWAL/RESCAN0000000002 entered state: DONE dummy-master,1,1 2013-10-04 14:31:13,908 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/splitWAL/RESCAN0000000002 2013-10-04 14:31:13,908 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager$DeleteAsyncCallback(1605): deleted /hbase/splitWAL/RESCAN0000000002 2013-10-04 14:31:13,908 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(943): deleted task without in memory state /hbase/splitWAL/RESCAN0000000002 2013-10-04 14:31:13,909 INFO [pool-1-thread-1] hbase.Waiter(174): Waiting up to [16,000] milli-secs(wait.for.ratio=[1]) 2013-10-04 14:31:18,337 INFO [dummy-master,1,1.splitLogManagerTimeoutMonitor] master.SplitLogManager$TimeoutMonitor(1408): total tasks = 2 unassigned = 2 tasks={/hbase/splitWAL/orphan%2F1=last_update = -1 last_version = 192 cur_worker_name = null status = in_progress incarnation = 1 resubmits = 1 batch = null, /hbase/splitWAL/foo%2F1=last_update = -1 last_version = -1 cur_worker_name = null status = in_progress incarnation = 0 resubmits = 0 batch = installed = 1 done = 0 error = 0} 2013-10-04 14:31:23,378 INFO [dummy-master,1,1.splitLogManagerTimeoutMonitor] master.SplitLogManager$TimeoutMonitor(1408): total tasks = 2 unassigned = 2 tasks={/hbase/splitWAL/orphan%2F1=last_update = -1 last_version = 192 cur_worker_name = null status = in_progress incarnation = 1 resubmits = 1 batch = null, /hbase/splitWAL/foo%2F1=last_update = -1 last_version = -1 cur_worker_name = null status = in_progress incarnation = 0 resubmits = 0 batch = installed = 1 done = 0 error = 0} 2013-10-04 14:31:25,998 DEBUG [dummy-master,1,1.splitLogManagerTimeoutMonitor] master.SplitLogManager$TimeoutMonitor(1440): resubmitting unassigned task(s) after timeout 2013-10-04 14:31:25,999 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(753): task not yet acquired /hbase/splitWAL/orphan%2F1 ver = 193 2013-10-04 14:31:25,999 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(753): task not yet acquired /hbase/splitWAL/foo%2F1 ver = 0 2013-10-04 14:31:26,000 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(761): task /hbase/splitWAL/RESCAN0000000003 entered state: DONE dummy-master,1,1 2013-10-04 14:31:26,001 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/splitWAL/RESCAN0000000003 2013-10-04 14:31:26,001 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager$DeleteAsyncCallback(1605): deleted /hbase/splitWAL/RESCAN0000000003 2013-10-04 14:31:26,001 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(943): deleted task without in memory state /hbase/splitWAL/RESCAN0000000003 2013-10-04 14:31:26,005 INFO [dummy-master,1,1.splitLogManagerTimeoutMonitor] hbase.Chore(93): dummy-master,1,1.splitLogManagerTimeoutMonitor exiting 2013-10-04 14:31:26,007 INFO [pool-1-thread-1] zookeeper.MiniZooKeeperCluster(249): Shutdown MiniZK cluster with all ZK servers 2013-10-04 14:31:26,081 INFO [pool-1-thread-1] hbase.ResourceChecker(171): after: master.TestSplitLogManager#testUnassignedTimeout Thread=29 (was 27) Potentially hanging thread: pool-1-thread-1-SendThread(localhost:49603) java.lang.Thread.sleep(Native Method) org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:219) org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1157) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1097) - Thread LEAK? -, OpenFileDescriptor=138 (was 150), MaxFileDescriptor=10240 (was 10240), SystemLoadAverage=703 (was 943), ProcessCount=260 (was 260), AvailableMemoryMB=3422 (was 3392) - AvailableMemoryMB LEAK? -, ConnectionCount=0 (was 0) 2013-10-04 14:31:26,108 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received ZooKeeper Event, type=None, state=Disconnected, path=null 2013-10-04 14:31:26,108 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(392): split-log-manager-testsd18d36b8-b099-419a-ab69-5b7dbe303375-0x14185614b250000 Received Disconnected from ZooKeeper, ignoring 2013-10-04 14:31:26,149 INFO [pool-1-thread-1] hbase.ResourceChecker(147): before: master.TestSplitLogManager#testTaskCreation Thread=29, OpenFileDescriptor=140, MaxFileDescriptor=10240, SystemLoadAverage=703, ProcessCount=260, AvailableMemoryMB=3423, ConnectionCount=0 2013-10-04 14:31:26,161 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(341): System.getProperty("hadoop.log.dir") already set to: /Users/stack/checkouts/hbase/hbase-server/target/test-data/beb7a788-857e-44ec-a61e-5b7716b9d10e/hadoop-log-dir so I do NOT create it in target/test-data/26bc830d-95b2-4a9c-b541-dbdb661aea58 2013-10-04 14:31:26,161 WARN [pool-1-thread-1] hbase.HBaseTestingUtility(345): hadoop.log.dir property value differs in configuration and system: Configuration=/Users/stack/checkouts/hbase/hbase-server/../logs while System=/Users/stack/checkouts/hbase/hbase-server/target/test-data/beb7a788-857e-44ec-a61e-5b7716b9d10e/hadoop-log-dir Erasing configuration value by system value. 2013-10-04 14:31:26,161 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(341): System.getProperty("hadoop.tmp.dir") already set to: /Users/stack/checkouts/hbase/hbase-server/target/test-data/beb7a788-857e-44ec-a61e-5b7716b9d10e/hadoop-tmp-dir so I do NOT create it in target/test-data/26bc830d-95b2-4a9c-b541-dbdb661aea58 2013-10-04 14:31:26,162 WARN [pool-1-thread-1] hbase.HBaseTestingUtility(345): hadoop.tmp.dir property value differs in configuration and system: Configuration=/tmp/hadoop-stack while System=/Users/stack/checkouts/hbase/hbase-server/target/test-data/beb7a788-857e-44ec-a61e-5b7716b9d10e/hadoop-tmp-dir Erasing configuration value by system value. 2013-10-04 14:31:26,162 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(398): Created new mini-cluster data directory: /Users/stack/checkouts/hbase/hbase-server/target/test-data/26bc830d-95b2-4a9c-b541-dbdb661aea58/dfscluster_7ae1652c-1133-4d4f-a51b-99790cfad29f, deleteOnExit=true 2013-10-04 14:31:26,169 INFO [pool-1-thread-1] zookeeper.MiniZooKeeperCluster(200): Started MiniZK Cluster and connect 1 ZK server on client port: 54504 2013-10-04 14:31:26,171 INFO [pool-1-thread-1] zookeeper.RecoverableZooKeeper(120): Process identifier=split-log-manager-tests2b0a682f-01c4-4a61-9890-3a30e50b68ee connecting to ZooKeeper ensemble=localhost:54504 2013-10-04 14:31:26,174 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-tests2b0a682f-01c4-4a61-9890-3a30e50b68ee Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2013-10-04 14:31:26,177 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(387): split-log-manager-tests2b0a682f-01c4-4a61-9890-3a30e50b68ee-0x1418561e0540000 connected 2013-10-04 14:31:26,179 DEBUG [pool-1-thread-1] master.TestSplitLogManager(119): /hbase created 2013-10-04 14:31:26,182 DEBUG [pool-1-thread-1] master.TestSplitLogManager(122): /hbase/splitWAL created 2013-10-04 14:31:26,182 INFO [pool-1-thread-1] master.TestSplitLogManager(199): TestTaskCreation - test the creation of a task in zk 2013-10-04 14:31:26,183 INFO [pool-1-thread-1] master.SplitLogManager(227): Timeout=6000, unassigned timeout=12000, distributedLogReplay=false 2013-10-04 14:31:26,184 INFO [pool-1-thread-1] master.SplitLogManager(1108): Found 0 orphan tasks and 0 rescan nodes 2013-10-04 14:31:26,185 DEBUG [pool-1-thread-1] zookeeper.ZKUtil(430): split-log-manager-tests2b0a682f-01c4-4a61-9890-3a30e50b68ee-0x1418561e0540000 Set watcher on znode that does not yet exist, /hbase/splitWAL/foo%2F1 2013-10-04 14:31:26,186 DEBUG [pool-1-thread-1] master.TestSplitLogManager(186): waiting for task node creation 2013-10-04 14:31:26,187 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-tests2b0a682f-01c4-4a61-9890-3a30e50b68ee-0x1418561e0540000 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/splitWAL/foo%2F1 2013-10-04 14:31:26,187 DEBUG [pool-1-thread-1-EventThread] regionserver.TestMasterAddressManager$NodeCreationListener(107): nodeCreated(/hbase/splitWAL/foo%2F1) 2013-10-04 14:31:26,187 DEBUG [pool-1-thread-1] master.TestSplitLogManager(188): task created 2013-10-04 14:31:26,187 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(711): put up splitlog task at znode /hbase/splitWAL/foo%2F1 2013-10-04 14:31:26,189 INFO [pool-1-thread-1] master.TestSplitLogManager(207): Task node created UNASSIGNED dummy-master,1,1 2013-10-04 14:31:26,189 INFO [dummy-master,1,1.splitLogManagerTimeoutMonitor] hbase.Chore(93): dummy-master,1,1.splitLogManagerTimeoutMonitor exiting 2013-10-04 14:31:26,190 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(753): task not yet acquired /hbase/splitWAL/foo%2F1 ver = 0 2013-10-04 14:31:26,194 INFO [pool-1-thread-1] zookeeper.MiniZooKeeperCluster(249): Shutdown MiniZK cluster with all ZK servers 2013-10-04 14:31:26,275 INFO [pool-1-thread-1] hbase.ResourceChecker(171): after: master.TestSplitLogManager#testTaskCreation Thread=32 (was 29) Potentially hanging thread: pool-1-thread-1-SendThread(localhost:54504) java.lang.Thread.sleep(Native Method) org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:219) org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1157) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1097) - Thread LEAK? -, OpenFileDescriptor=146 (was 140) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=10240 (was 10240), SystemLoadAverage=703 (was 703), ProcessCount=260 (was 260), AvailableMemoryMB=3433 (was 3423) - AvailableMemoryMB LEAK? -, ConnectionCount=0 (was 0) 2013-10-04 14:31:26,293 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-tests2b0a682f-01c4-4a61-9890-3a30e50b68ee-0x1418561e0540000 Received ZooKeeper Event, type=None, state=Disconnected, path=null 2013-10-04 14:31:26,293 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(392): split-log-manager-tests2b0a682f-01c4-4a61-9890-3a30e50b68ee-0x1418561e0540000 Received Disconnected from ZooKeeper, ignoring 2013-10-04 14:31:26,355 INFO [pool-1-thread-1] hbase.ResourceChecker(147): before: master.TestSplitLogManager#testDeadWorker Thread=32, OpenFileDescriptor=148, MaxFileDescriptor=10240, SystemLoadAverage=703, ProcessCount=258, AvailableMemoryMB=3492, ConnectionCount=0 2013-10-04 14:31:26,366 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(341): System.getProperty("hadoop.log.dir") already set to: /Users/stack/checkouts/hbase/hbase-server/target/test-data/beb7a788-857e-44ec-a61e-5b7716b9d10e/hadoop-log-dir so I do NOT create it in target/test-data/834c1918-c27f-47e4-acdc-243c21213c37 2013-10-04 14:31:26,367 WARN [pool-1-thread-1] hbase.HBaseTestingUtility(345): hadoop.log.dir property value differs in configuration and system: Configuration=/Users/stack/checkouts/hbase/hbase-server/../logs while System=/Users/stack/checkouts/hbase/hbase-server/target/test-data/beb7a788-857e-44ec-a61e-5b7716b9d10e/hadoop-log-dir Erasing configuration value by system value. 2013-10-04 14:31:26,367 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(341): System.getProperty("hadoop.tmp.dir") already set to: /Users/stack/checkouts/hbase/hbase-server/target/test-data/beb7a788-857e-44ec-a61e-5b7716b9d10e/hadoop-tmp-dir so I do NOT create it in target/test-data/834c1918-c27f-47e4-acdc-243c21213c37 2013-10-04 14:31:26,367 WARN [pool-1-thread-1] hbase.HBaseTestingUtility(345): hadoop.tmp.dir property value differs in configuration and system: Configuration=/tmp/hadoop-stack while System=/Users/stack/checkouts/hbase/hbase-server/target/test-data/beb7a788-857e-44ec-a61e-5b7716b9d10e/hadoop-tmp-dir Erasing configuration value by system value. 2013-10-04 14:31:26,367 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(398): Created new mini-cluster data directory: /Users/stack/checkouts/hbase/hbase-server/target/test-data/834c1918-c27f-47e4-acdc-243c21213c37/dfscluster_11376668-9229-42ef-8237-5747650d4747, deleteOnExit=true 2013-10-04 14:31:26,374 INFO [pool-1-thread-1] zookeeper.MiniZooKeeperCluster(200): Started MiniZK Cluster and connect 1 ZK server on client port: 58559 2013-10-04 14:31:26,376 INFO [pool-1-thread-1] zookeeper.RecoverableZooKeeper(120): Process identifier=split-log-manager-testsde08cd05-96eb-4100-9797-50aa8aca78d9 connecting to ZooKeeper ensemble=localhost:58559 2013-10-04 14:31:26,380 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsde08cd05-96eb-4100-9797-50aa8aca78d9 Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2013-10-04 14:31:26,383 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(387): split-log-manager-testsde08cd05-96eb-4100-9797-50aa8aca78d9-0x1418561e1210000 connected 2013-10-04 14:31:26,386 DEBUG [pool-1-thread-1] master.TestSplitLogManager(119): /hbase created 2013-10-04 14:31:26,389 DEBUG [pool-1-thread-1] master.TestSplitLogManager(122): /hbase/splitWAL created 2013-10-04 14:31:26,390 INFO [pool-1-thread-1] master.TestSplitLogManager(432): testDeadWorker 2013-10-04 14:31:26,390 INFO [pool-1-thread-1] master.SplitLogManager(227): Timeout=6000, unassigned timeout=12000, distributedLogReplay=false 2013-10-04 14:31:26,391 INFO [pool-1-thread-1] master.SplitLogManager(1108): Found 0 orphan tasks and 0 rescan nodes 2013-10-04 14:31:26,392 DEBUG [pool-1-thread-1] zookeeper.ZKUtil(430): split-log-manager-testsde08cd05-96eb-4100-9797-50aa8aca78d9-0x1418561e1210000 Set watcher on znode that does not yet exist, /hbase/splitWAL/foo%2F1 2013-10-04 14:31:26,393 DEBUG [pool-1-thread-1] master.TestSplitLogManager(186): waiting for task node creation 2013-10-04 14:31:26,394 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsde08cd05-96eb-4100-9797-50aa8aca78d9-0x1418561e1210000 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/splitWAL/foo%2F1 2013-10-04 14:31:26,394 DEBUG [pool-1-thread-1-EventThread] regionserver.TestMasterAddressManager$NodeCreationListener(107): nodeCreated(/hbase/splitWAL/foo%2F1) 2013-10-04 14:31:26,394 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(711): put up splitlog task at znode /hbase/splitWAL/foo%2F1 2013-10-04 14:31:26,394 DEBUG [pool-1-thread-1] master.TestSplitLogManager(188): task created 2013-10-04 14:31:26,395 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(753): task not yet acquired /hbase/splitWAL/foo%2F1 ver = 0 2013-10-04 14:31:26,397 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsde08cd05-96eb-4100-9797-50aa8aca78d9-0x1418561e1210000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/foo%2F1 2013-10-04 14:31:26,397 INFO [pool-1-thread-1] hbase.Waiter(174): Waiting up to [3,200] milli-secs(wait.for.ratio=[1]) 2013-10-04 14:31:26,398 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(826): task /hbase/splitWAL/foo%2F1 acquired by worker1,1,1 2013-10-04 14:31:26,409 INFO [pool-1-thread-1] master.SplitLogManager(1339): dead splitlog worker worker1,1,1 2013-10-04 14:31:26,409 INFO [pool-1-thread-1] hbase.Waiter(174): Waiting up to [9,600] milli-secs(wait.for.ratio=[1]) 2013-10-04 14:31:26,492 INFO [dummy-master,1,1.splitLogManagerTimeoutMonitor] master.SplitLogManager(875): resubmitting task /hbase/splitWAL/foo%2F1 2013-10-04 14:31:26,494 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsde08cd05-96eb-4100-9797-50aa8aca78d9-0x1418561e1210000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/foo%2F1 2013-10-04 14:31:26,494 INFO [dummy-master,1,1.splitLogManagerTimeoutMonitor] master.SplitLogManager$TimeoutMonitor(1408): total tasks = 1 unassigned = 0 tasks={/hbase/splitWAL/foo%2F1=last_update = -1 last_version = 1 cur_worker_name = null status = in_progress incarnation = 1 resubmits = 0 batch = installed = 1 done = 0 error = 0} 2013-10-04 14:31:26,494 INFO [dummy-master,1,1.splitLogManagerTimeoutMonitor] master.SplitLogManager$TimeoutMonitor(1412): resubmitted 1 out of 1 tasks 2013-10-04 14:31:26,495 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(753): task not yet acquired /hbase/splitWAL/foo%2F1 ver = 2 2013-10-04 14:31:26,497 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(761): task /hbase/splitWAL/RESCAN0000000001 entered state: DONE dummy-master,1,1 2013-10-04 14:31:26,498 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsde08cd05-96eb-4100-9797-50aa8aca78d9-0x1418561e1210000 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/splitWAL/RESCAN0000000001 2013-10-04 14:31:26,498 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager$DeleteAsyncCallback(1605): deleted /hbase/splitWAL/RESCAN0000000001 2013-10-04 14:31:26,498 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(943): deleted task without in memory state /hbase/splitWAL/RESCAN0000000001 2013-10-04 14:31:26,499 INFO [dummy-master,1,1.splitLogManagerTimeoutMonitor] hbase.Chore(93): dummy-master,1,1.splitLogManagerTimeoutMonitor exiting 2013-10-04 14:31:26,500 INFO [pool-1-thread-1] zookeeper.MiniZooKeeperCluster(249): Shutdown MiniZK cluster with all ZK servers 2013-10-04 14:31:26,580 INFO [pool-1-thread-1] hbase.ResourceChecker(171): after: master.TestSplitLogManager#testDeadWorker Thread=35 (was 32) Potentially hanging thread: pool-1-thread-1-SendThread(localhost:58559) java.lang.Thread.sleep(Native Method) org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:219) org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1157) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1097) - Thread LEAK? -, OpenFileDescriptor=154 (was 148) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=10240 (was 10240), SystemLoadAverage=703 (was 703), ProcessCount=260 (was 258) - ProcessCount LEAK? -, AvailableMemoryMB=3472 (was 3492), ConnectionCount=0 (was 0) 2013-10-04 14:31:26,600 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsde08cd05-96eb-4100-9797-50aa8aca78d9-0x1418561e1210000 Received ZooKeeper Event, type=None, state=Disconnected, path=null 2013-10-04 14:31:26,600 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(392): split-log-manager-testsde08cd05-96eb-4100-9797-50aa8aca78d9-0x1418561e1210000 Received Disconnected from ZooKeeper, ignoring 2013-10-04 14:31:26,672 INFO [pool-1-thread-1] hbase.ResourceChecker(147): before: master.TestSplitLogManager#testRescanCleanup Thread=35, OpenFileDescriptor=156, MaxFileDescriptor=10240, SystemLoadAverage=703, ProcessCount=260, AvailableMemoryMB=3467, ConnectionCount=0 2013-10-04 14:31:26,685 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(341): System.getProperty("hadoop.log.dir") already set to: /Users/stack/checkouts/hbase/hbase-server/target/test-data/beb7a788-857e-44ec-a61e-5b7716b9d10e/hadoop-log-dir so I do NOT create it in target/test-data/d2f2d19f-a5f8-4b33-bc2a-5001a91f5b20 2013-10-04 14:31:26,685 WARN [pool-1-thread-1] hbase.HBaseTestingUtility(345): hadoop.log.dir property value differs in configuration and system: Configuration=/Users/stack/checkouts/hbase/hbase-server/../logs while System=/Users/stack/checkouts/hbase/hbase-server/target/test-data/beb7a788-857e-44ec-a61e-5b7716b9d10e/hadoop-log-dir Erasing configuration value by system value. 2013-10-04 14:31:26,685 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(341): System.getProperty("hadoop.tmp.dir") already set to: /Users/stack/checkouts/hbase/hbase-server/target/test-data/beb7a788-857e-44ec-a61e-5b7716b9d10e/hadoop-tmp-dir so I do NOT create it in target/test-data/d2f2d19f-a5f8-4b33-bc2a-5001a91f5b20 2013-10-04 14:31:26,686 WARN [pool-1-thread-1] hbase.HBaseTestingUtility(345): hadoop.tmp.dir property value differs in configuration and system: Configuration=/tmp/hadoop-stack while System=/Users/stack/checkouts/hbase/hbase-server/target/test-data/beb7a788-857e-44ec-a61e-5b7716b9d10e/hadoop-tmp-dir Erasing configuration value by system value. 2013-10-04 14:31:26,686 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(398): Created new mini-cluster data directory: /Users/stack/checkouts/hbase/hbase-server/target/test-data/d2f2d19f-a5f8-4b33-bc2a-5001a91f5b20/dfscluster_263a84d2-597c-474f-962f-ada1f46dcd6c, deleteOnExit=true 2013-10-04 14:31:26,696 INFO [pool-1-thread-1] zookeeper.MiniZooKeeperCluster(200): Started MiniZK Cluster and connect 1 ZK server on client port: 49837 2013-10-04 14:31:26,699 INFO [pool-1-thread-1] zookeeper.RecoverableZooKeeper(120): Process identifier=split-log-manager-testsb39e1094-7e70-41ce-ae22-5b09c6248fa2 connecting to ZooKeeper ensemble=localhost:49837 2013-10-04 14:31:26,704 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsb39e1094-7e70-41ce-ae22-5b09c6248fa2 Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2013-10-04 14:31:26,707 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(387): split-log-manager-testsb39e1094-7e70-41ce-ae22-5b09c6248fa2-0x1418561e2620000 connected 2013-10-04 14:31:26,709 DEBUG [pool-1-thread-1] master.TestSplitLogManager(119): /hbase created 2013-10-04 14:31:26,712 DEBUG [pool-1-thread-1] master.TestSplitLogManager(122): /hbase/splitWAL created 2013-10-04 14:31:26,713 INFO [pool-1-thread-1] master.TestSplitLogManager(299): TestRescanCleanup - ensure RESCAN nodes are cleaned up 2013-10-04 14:31:26,713 INFO [pool-1-thread-1] master.SplitLogManager(227): Timeout=6000, unassigned timeout=12000, distributedLogReplay=false 2013-10-04 14:31:26,715 INFO [pool-1-thread-1] master.SplitLogManager(1108): Found 0 orphan tasks and 0 rescan nodes 2013-10-04 14:31:26,716 DEBUG [pool-1-thread-1] zookeeper.ZKUtil(430): split-log-manager-testsb39e1094-7e70-41ce-ae22-5b09c6248fa2-0x1418561e2620000 Set watcher on znode that does not yet exist, /hbase/splitWAL/foo%2F1 2013-10-04 14:31:26,717 DEBUG [pool-1-thread-1] master.TestSplitLogManager(186): waiting for task node creation 2013-10-04 14:31:26,718 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsb39e1094-7e70-41ce-ae22-5b09c6248fa2-0x1418561e2620000 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/splitWAL/foo%2F1 2013-10-04 14:31:26,718 DEBUG [pool-1-thread-1-EventThread] regionserver.TestMasterAddressManager$NodeCreationListener(107): nodeCreated(/hbase/splitWAL/foo%2F1) 2013-10-04 14:31:26,718 DEBUG [pool-1-thread-1] master.TestSplitLogManager(188): task created 2013-10-04 14:31:26,718 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(711): put up splitlog task at znode /hbase/splitWAL/foo%2F1 2013-10-04 14:31:26,719 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(753): task not yet acquired /hbase/splitWAL/foo%2F1 ver = 0 2013-10-04 14:31:26,720 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsb39e1094-7e70-41ce-ae22-5b09c6248fa2-0x1418561e2620000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/foo%2F1 2013-10-04 14:31:26,721 INFO [pool-1-thread-1] hbase.Waiter(174): Waiting up to [3,200] milli-secs(wait.for.ratio=[1]) 2013-10-04 14:31:26,721 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(826): task /hbase/splitWAL/foo%2F1 acquired by worker1,1,1 2013-10-04 14:31:26,733 INFO [pool-1-thread-1] hbase.Waiter(174): Waiting up to [300,000] milli-secs(wait.for.ratio=[1]) 2013-10-04 14:31:26,816 INFO [dummy-master,1,1.splitLogManagerTimeoutMonitor] master.SplitLogManager$TimeoutMonitor(1408): total tasks = 1 unassigned = 0 tasks={/hbase/splitWAL/foo%2F1=last_update = 1380922286721 last_version = 1 cur_worker_name = worker1,1,1 status = in_progress incarnation = 0 resubmits = 0 batch = installed = 1 done = 0 error = 0} 2013-10-04 14:31:31,853 INFO [dummy-master,1,1.splitLogManagerTimeoutMonitor] master.SplitLogManager$TimeoutMonitor(1408): total tasks = 1 unassigned = 0 tasks={/hbase/splitWAL/foo%2F1=last_update = 1380922286721 last_version = 1 cur_worker_name = worker1,1,1 status = in_progress incarnation = 0 resubmits = 0 batch = installed = 1 done = 0 error = 0} 2013-10-04 14:31:32,761 INFO [dummy-master,1,1.splitLogManagerTimeoutMonitor] master.SplitLogManager(875): resubmitting task /hbase/splitWAL/foo%2F1 2013-10-04 14:31:32,764 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsb39e1094-7e70-41ce-ae22-5b09c6248fa2-0x1418561e2620000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/foo%2F1 2013-10-04 14:31:32,765 INFO [dummy-master,1,1.splitLogManagerTimeoutMonitor] master.SplitLogManager$TimeoutMonitor(1412): resubmitted 1 out of 1 tasks 2013-10-04 14:31:32,766 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(753): task not yet acquired /hbase/splitWAL/foo%2F1 ver = 2 2013-10-04 14:31:32,767 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(761): task /hbase/splitWAL/RESCAN0000000001 entered state: DONE dummy-master,1,1 2013-10-04 14:31:32,769 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsb39e1094-7e70-41ce-ae22-5b09c6248fa2-0x1418561e2620000 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/splitWAL/RESCAN0000000001 2013-10-04 14:31:32,769 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager$DeleteAsyncCallback(1605): deleted /hbase/splitWAL/RESCAN0000000001 2013-10-04 14:31:32,769 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(943): deleted task without in memory state /hbase/splitWAL/RESCAN0000000001 2013-10-04 14:31:32,776 INFO [pool-1-thread-1] hbase.Waiter(174): Waiting up to [3,200] milli-secs(wait.for.ratio=[1]) 2013-10-04 14:31:32,777 INFO [dummy-master,1,1.splitLogManagerTimeoutMonitor] hbase.Chore(93): dummy-master,1,1.splitLogManagerTimeoutMonitor exiting 2013-10-04 14:31:32,782 INFO [pool-1-thread-1] zookeeper.MiniZooKeeperCluster(249): Shutdown MiniZK cluster with all ZK servers 2013-10-04 14:31:32,879 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(310): split-log-manager-testsb39e1094-7e70-41ce-ae22-5b09c6248fa2-0x1418561e2620000 Received ZooKeeper Event, type=None, state=Disconnected, path=null 2013-10-04 14:31:32,879 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(392): split-log-manager-testsb39e1094-7e70-41ce-ae22-5b09c6248fa2-0x1418561e2620000 Received Disconnected from ZooKeeper, ignoring 2013-10-04 14:31:32,895 INFO [pool-1-thread-1] hbase.ResourceChecker(171): after: master.TestSplitLogManager#testRescanCleanup Thread=35 (was 35), OpenFileDescriptor=162 (was 156) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=10240 (was 10240), SystemLoadAverage=664 (was 703), ProcessCount=260 (was 260), AvailableMemoryMB=3467 (was 3467), ConnectionCount=0 (was 0)