| Regressionorg.apache.hadoop.hbase.replication.TestReplicationSmallTests.testDisableEnable Failing for the past 1 build
(Since #444 )
Error MessageWaited too much time for put replication Stacktracejava.lang.AssertionError: Waited too much time for put replication
at org.junit.Assert.fail(Assert.java:88)
at org.apache.hadoop.hbase.replication.TestReplicationSmallTests.testDisableEnable(TestReplicationSmallTests.java:313)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
Standard Output2013-08-14 08:04:58,201 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(354): Created new mini-cluster data directory: /home/hudson/jenkins-slave/workspace/hbase-0.95/0.95/hbase-server/target/test-data/b0bbd7fb-86d5-413b-86b1-c0a381722f5f/dfscluster_ad5403a2-ad24-41cc-b770-9798c1d50a66
2013-08-14 08:04:58,372 INFO [pool-1-thread-1] zookeeper.MiniZooKeeperCluster(197): Started MiniZK Cluster and connect 1 ZK server on client port: 64039
2013-08-14 08:04:58,422 INFO [pool-1-thread-1] zookeeper
...[truncated 2073012 bytes]...
p.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:428)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.nodeDeleted(ZooKeeperNodeTracker.java:211)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:336)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
2013-08-14 08:07:12,592 INFO [pool-1-thread-1-EventThread] master.HMaster(2409): Aborting
Standard Error2013-08-14 08:06:43,252 INFO [pool-1-thread-1] hbase.ResourceChecker(147): before: replication.TestReplicationSmallTests#testDisableEnable Thread=597, OpenFileDescriptor=893, MaxFileDescriptor=4096, SystemLoadAverage=792, ProcessCount=135, AvailableMemoryMB=442, ConnectionCount=9
2013-08-14 08:06:43,252 WARN [pool-1-thread-1] hbase.ResourceChecker(134): Thread=597 is superior to 500
2013-08-14 08:06:43,267 DEBUG [RS:1;vesta:39079-EventThread.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/jenkins/hbase/WALs/vesta.apache.org,39079,1376467506138/vesta.apache.org%2C39079%2C1376467506138.1376467601635 is an HLog file, so reordering blocks, last hostname will be:vesta.apache.org
2013-08-14 08:06:43,390 INFO [vesta:43149Replication Statistics #0] regionserver.Replication$ReplicationStatisticsThread(324): Normal source for cluster 2: Total replicated edits: 2, currently replicating from: hdfs://localhost:52881/user/jenkins/hbase/WALs/vesta.apache.org,43149,1376467506109/vesta.apache.org%2C43149%2C1376467506109.1376467569820 at position: 0
2013-08-14 08:06:43,390 INFO [vesta:39079Replication Statistics #0] regionserver.Replication$ReplicationStatisticsThread(324): Normal source for cluster 2: Total replicated edits: 104, currently replicating from: hdfs://localhost:52881/user/jenkins/hbase/WALs/vesta.apache.org,39079,1376467506138/vesta.apache.org%2C39079%2C1376467506138.1376467601635 at position: 15767
2013-08-14 08:06:44,179 DEBUG [RS:0;vesta:43149-EventThread.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/jenkins/hbase/WALs/vesta.apache.org,43149,1376467506109/vesta.apache.org%2C43149%2C1376467506109.1376467569820 is an HLog file, so reordering blocks, last hostname will be:vesta.apache.org
2013-08-14 08:06:44,180 DEBUG [RS:0;vesta:43149-EventThread.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/jenkins/hbase/WALs/vesta.apache.org,43149,1376467506109/vesta.apache.org%2C43149%2C1376467506109.1376467569820 is an HLog file, so reordering blocks, last hostname will be:vesta.apache.org
2013-08-14 08:06:44,180 DEBUG [RS:0;vesta:43149-EventThread.replicationSource,2] regionserver.ReplicationSource(578): Unable to open a reader, sleeping 100 times 10
2013-08-14 08:06:44,214 INFO [vesta:37982Replication Statistics #0] regionserver.Replication$ReplicationStatisticsThread(324): Sink: age in ms of last applied edit: 40427, total replicated edits: 2
2013-08-14 08:06:44,302 INFO [vesta:36402Replication Statistics #0] regionserver.Replication$ReplicationStatisticsThread(324): Sink: age in ms of last applied edit: 1647, total replicated edits: 1359
2013-08-14 08:06:45,181 DEBUG [RS:0;vesta:43149-EventThread.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/jenkins/hbase/WALs/vesta.apache.org,43149,1376467506109/vesta.apache.org%2C43149%2C1376467506109.1376467569820 is an HLog file, so reordering blocks, last hostname will be:vesta.apache.org
2013-08-14 08:06:45,183 DEBUG [RS:0;vesta:43149-EventThread.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/jenkins/hbase/WALs/vesta.apache.org,43149,1376467506109/vesta.apache.org%2C43149%2C1376467506109.1376467569820 is an HLog file, so reordering blocks, last hostname will be:vesta.apache.org
2013-08-14 08:06:45,183 DEBUG [RS:0;vesta:43149-EventThread.replicationSource,2] regionserver.ReplicationSource(578): Unable to open a reader, sleeping 100 times 10
2013-08-14 08:06:46,184 DEBUG [RS:0;vesta:43149-EventThread.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/jenkins/hbase/WALs/vesta.apache.org,43149,1376467506109/vesta.apache.org%2C43149%2C1376467506109.1376467569820 is an HLog file, so reordering blocks, last hostname will be:vesta.apache.org
2013-08-14 08:06:46,185 DEBUG [RS:0;vesta:43149-EventThread.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/jenkins/hbase/WALs/vesta.apache.org,43149,1376467506109/vesta.apache.org%2C43149%2C1376467506109.1376467569820 is an HLog file, so reordering blocks, last hostname will be:vesta.apache.org
2013-08-14 08:06:46,186 DEBUG [RS:0;vesta:43149-EventThread.replicationSource,2] regionserver.ReplicationSource(578): Unable to open a reader, sleeping 100 times 10
2013-08-14 08:06:46,799 DEBUG [RS:1;vesta:39079-EventThread.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false
2013-08-14 08:06:46,799 DEBUG [RS:1;vesta:39079-EventThread.replicationSource,2] regionserver.ReplicationSource(578): Nothing to replicate, sleeping 100 times 1
2013-08-14 08:06:46,900 DEBUG [RS:1;vesta:39079-EventThread.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/jenkins/hbase/WALs/vesta.apache.org,39079,1376467506138/vesta.apache.org%2C39079%2C1376467506138.1376467601635 is an HLog file, so reordering blocks, last hostname will be:vesta.apache.org
2013-08-14 08:06:46,903 DEBUG [RS:1;vesta:39079-EventThread.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 15767, fileLength: 15775, trailerPresent: true
2013-08-14 08:06:46,903 DEBUG [RS:1;vesta:39079-EventThread.replicationSource,2] regionserver.ReplicationSource(578): Nothing to replicate, sleeping 100 times 1
2013-08-14 08:06:47,004 DEBUG [RS:1;vesta:39079-EventThread.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/jenkins/hbase/WALs/vesta.apache.org,39079,1376467506138/vesta.apache.org%2C39079%2C1376467506138.1376467601635 is an HLog file, so reordering blocks, last hostname will be:vesta.apache.org
2013-08-14 08:06:47,091 DEBUG [RS:1;vesta:39079-EventThread.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 15767, fileLength: 15775, trailerPresent: true
2013-08-14 08:06:47,091 INFO [pool-1-thread-1] wal.FSHLog(523): Rolled WAL /user/jenkins/hbase/WALs/vesta.apache.org,39079,1376467506138/vesta.apache.org%2C39079%2C1376467506138.1376467601635 with entries=101, filesize=15.4k; new WAL /user/jenkins/hbase/WALs/vesta.apache.org,39079,1376467506138/vesta.apache.org%2C39079%2C1376467506138.1376467603252
2013-08-14 08:06:47,091 DEBUG [RS:1;vesta:39079-EventThread.replicationSource,2] regionserver.ReplicationSource(578): Nothing to replicate, sleeping 100 times 1
2013-08-14 08:06:47,091 INFO [pool-1-thread-1] wal.FSHLog(636): Too many hlogs: logs=12, maxlogs=10; forcing flush of 1 regions(s): 4826936a93ccdf218bf4ff65c767792d
2013-08-14 08:06:47,186 DEBUG [RS:0;vesta:43149-EventThread.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/jenkins/hbase/WALs/vesta.apache.org,43149,1376467506109/vesta.apache.org%2C43149%2C1376467506109.1376467569820 is an HLog file, so reordering blocks, last hostname will be:vesta.apache.org
2013-08-14 08:06:47,188 DEBUG [RS:0;vesta:43149-EventThread.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/jenkins/hbase/WALs/vesta.apache.org,43149,1376467506109/vesta.apache.org%2C43149%2C1376467506109.1376467569820 is an HLog file, so reordering blocks, last hostname will be:vesta.apache.org
2013-08-14 08:06:47,188 DEBUG [RS:0;vesta:43149-EventThread.replicationSource,2] regionserver.ReplicationSource(578): Unable to open a reader, sleeping 100 times 10
2013-08-14 08:06:47,192 DEBUG [RS:1;vesta:39079-EventThread.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/jenkins/hbase/WALs/vesta.apache.org,39079,1376467506138/vesta.apache.org%2C39079%2C1376467506138.1376467601635 is an HLog file, so reordering blocks, last hostname will be:vesta.apache.org
2013-08-14 08:06:47,194 DEBUG [RS:1;vesta:39079-EventThread.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 15767, fileLength: 15775, trailerPresent: true
2013-08-14 08:06:47,200 DEBUG [RS:1;vesta:39079-EventThread.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/jenkins/hbase/WALs/vesta.apache.org,39079,1376467506138/vesta.apache.org%2C39079%2C1376467506138.1376467603252 is an HLog file, so reordering blocks, last hostname will be:vesta.apache.org
2013-08-14 08:06:47,203 DEBUG [RS:1;vesta:39079-EventThread.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false
2013-08-14 08:06:47,226 DEBUG [RS:1;vesta:39079-EventThread.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/jenkins/hbase/WALs/vesta.apache.org,39079,1376467506138/vesta.apache.org%2C39079%2C1376467506138.1376467603252 is an HLog file, so reordering blocks, last hostname will be:vesta.apache.org
2013-08-14 08:06:47,228 DEBUG [RS:1;vesta:39079-EventThread.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false
2013-08-14 08:06:47,228 WARN [RS:1;vesta:39079-EventThread.replicationSource,2] regionserver.ReplicationSource(301): 2 Got:
java.io.IOException: Cannot seek after EOF
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.seek(DFSClient.java:2593)
at org.apache.hadoop.fs.FSDataInputStream.seek(FSDataInputStream.java:37)
at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.seekOnFs(ProtobufLogReader.java:275)
at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.seek(ReaderBase.java:115)
at org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.seek(ReplicationHLogReaderManager.java:108)
at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.readAllEntriesToReplicateOrNextFile(ReplicationSource.java:388)
at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:297)
2013-08-14 08:06:47,228 DEBUG [RS:1;vesta:39079-EventThread.replicationSource,2] regionserver.ReplicationSource(578): Nothing to replicate, sleeping 100 times 1
2013-08-14 08:06:47,329 DEBUG [RS:1;vesta:39079-EventThread.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/jenkins/hbase/WALs/vesta.apache.org,39079,1376467506138/vesta.apache.org%2C39079%2C1376467506138.1376467603252 is an HLog file, so reordering blocks, last hostname will be:vesta.apache.org
2013-08-14 08:06:47,331 DEBUG [RS:1;vesta:39079-EventThread.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false
2013-08-14 08:06:47,332 DEBUG [RS:1;vesta:39079-EventThread.replicationSource,2] regionserver.ReplicationSource(578): Nothing to replicate, sleeping 100 times 1
2013-08-14 08:06:47,432 DEBUG [RS:1;vesta:39079-EventThread.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/jenkins/hbase/WALs/vesta.apache.org,39079,1376467506138/vesta.apache.org%2C39079%2C1376467506138.1376467603252 is an HLog file, so reordering blocks, last hostname will be:vesta.apache.org
2013-08-14 08:06:47,435 DEBUG [RS:1;vesta:39079-EventThread.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false
2013-08-14 08:06:47,451 DEBUG [RS:1;vesta:39079-EventThread.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/jenkins/hbase/WALs/vesta.apache.org,39079,1376467506138/vesta.apache.org%2C39079%2C1376467506138.1376467603252 is an HLog file, so reordering blocks, last hostname will be:vesta.apache.org
2013-08-14 08:06:47,454 DEBUG [RS:1;vesta:39079-EventThread.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false
2013-08-14 08:06:47,467 DEBUG [RS:1;vesta:39079-EventThread.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/jenkins/hbase/WALs/vesta.apache.org,39079,1376467506138/vesta.apache.org%2C39079%2C1376467506138.1376467603252 is an HLog file, so reordering blocks, last hostname will be:vesta.apache.org
2013-08-14 08:06:47,470 DEBUG [RS:1;vesta:39079-EventThread.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false
2013-08-14 08:06:47,483 DEBUG [RS:1;vesta:39079-EventThread.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/jenkins/hbase/WALs/vesta.apache.org,39079,1376467506138/vesta.apache.org%2C39079%2C1376467506138.1376467603252 is an HLog file, so reordering blocks, last hostname will be:vesta.apache.org
2013-08-14 08:06:47,486 DEBUG [RS:1;vesta:39079-EventThread.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false
2013-08-14 08:06:47,507 DEBUG [RS:1;vesta:39079-EventThread.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/jenkins/hbase/WALs/vesta.apache.org,39079,1376467506138/vesta.apache.org%2C39079%2C1376467506138.1376467603252 is an HLog file, so reordering blocks, last hostname will be:vesta.apache.org
2013-08-14 08:06:47,510 DEBUG [RS:1;vesta:39079-EventThread.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false
2013-08-14 08:06:47,583 DEBUG [RS:1;vesta:39079-EventThread.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/jenkins/hbase/WALs/vesta.apache.org,39079,1376467506138/vesta.apache.org%2C39079%2C1376467506138.1376467603252 is an HLog file, so reordering blocks, last hostname will be:vesta.apache.org
2013-08-14 08:06:47,585 DEBUG [RS:1;vesta:39079-EventThread.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false
2013-08-14 08:06:47,585 DEBUG [RS:1;vesta:39079-EventThread.replicationSource,2] regionserver.ReplicationSource(578): Nothing to replicate, sleeping 100 times 1
2013-08-14 08:06:47,686 DEBUG [RS:1;vesta:39079-EventThread.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/jenkins/hbase/WALs/vesta.apache.org,39079,1376467506138/vesta.apache.org%2C39079%2C1376467506138.1376467603252 is an HLog file, so reordering blocks, last hostname will be:vesta.apache.org
2013-08-14 08:06:47,689 DEBUG [RS:1;vesta:39079-EventThread.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false
2013-08-14 08:06:47,695 DEBUG [pool-1-thread-1] client.ClientScanner(224): Finished with region {ENCODED => 4826936a93ccdf218bf4ff65c767792d, NAME => 'test,,1376467521631.4826936a93ccdf218bf4ff65c767792d.', STARTKEY => '', ENDKEY => 'bbb'}
2013-08-14 08:06:47,695 DEBUG [pool-1-thread-1] client.ClientScanner(232): Advancing internal scanner to startKey at 'bbb'
2013-08-14 08:06:47,697 DEBUG [pool-1-thread-1] client.ClientScanner(224): Finished with region {ENCODED => f1e60e707f34cf0ab715adb72be3948f, NAME => 'test,bbb,1376467521631.f1e60e707f34cf0ab715adb72be3948f.', STARTKEY => 'bbb', ENDKEY => 'ccc'}
2013-08-14 08:06:47,697 DEBUG [pool-1-thread-1] client.ClientScanner(232): Advancing internal scanner to startKey at 'ccc'
2013-08-14 08:06:47,698 DEBUG [pool-1-thread-1] client.ClientScanner(224): Finished with region {ENCODED => 2080e59bfd9ff5049e0f1dfece187e2c, NAME => 'test,ccc,1376467521631.2080e59bfd9ff5049e0f1dfece187e2c.', STARTKEY => 'ccc', ENDKEY => 'ddd'}
2013-08-14 08:06:47,698 DEBUG [pool-1-thread-1] client.ClientScanner(232): Advancing internal scanner to startKey at 'ddd'
2013-08-14 08:06:47,700 DEBUG [pool-1-thread-1] client.ClientScanner(224): Finished with region {ENCODED => 9fed0033503397b953a9cb3daeae5cd4, NAME => 'test,ddd,1376467521631.9fed0033503397b953a9cb3daeae5cd4.', STARTKEY => 'ddd', ENDKEY => 'eee'}
2013-08-14 08:06:47,700 DEBUG [pool-1-thread-1] client.ClientScanner(232): Advancing internal scanner to startKey at 'eee'
2013-08-14 08:06:47,701 DEBUG [pool-1-thread-1] client.ClientScanner(224): Finished with region {ENCODED => 912081c13b971d6d5b6810a2b57b4570, NAME => 'test,eee,1376467521631.912081c13b971d6d5b6810a2b57b4570.', STARTKEY => 'eee', ENDKEY => 'fff'}
2013-08-14 08:06:47,701 DEBUG [pool-1-thread-1] client.ClientScanner(232): Advancing internal scanner to startKey at 'fff'
2013-08-14 08:06:47,703 DEBUG [pool-1-thread-1] client.ClientScanner(224): Finished with region {ENCODED => dd1806f1ff95bb4f3896c769f58a39cd, NAME => 'test,fff,1376467521631.dd1806f1ff95bb4f3896c769f58a39cd.', STARTKEY => 'fff', ENDKEY => 'ggg'}
2013-08-14 08:06:47,703 DEBUG [pool-1-thread-1] client.ClientScanner(232): Advancing internal scanner to startKey at 'ggg'
2013-08-14 08:06:47,704 DEBUG [pool-1-thread-1] client.ClientScanner(224): Finished with region {ENCODED => 0e8e78b5503a9d7e2b10084a09b8a177, NAME => 'test,ggg,1376467521631.0e8e78b5503a9d7e2b10084a09b8a177.', STARTKEY => 'ggg', ENDKEY => 'hhh'}
2013-08-14 08:06:47,704 DEBUG [pool-1-thread-1] client.ClientScanner(232): Advancing internal scanner to startKey at 'hhh'
2013-08-14 08:06:47,705 DEBUG [pool-1-thread-1] client.ClientScanner(224): Finished with region {ENCODED => 8d762c270df8abcc9752e0c5f4cf06b5, NAME => 'test,hhh,1376467521631.8d762c270df8abcc9752e0c5f4cf06b5.', STARTKEY => 'hhh', ENDKEY => 'iii'}
2013-08-14 08:06:47,705 DEBUG [pool-1-thread-1] client.ClientScanner(232): Advancing internal scanner to startKey at 'iii'
2013-08-14 08:06:47,707 DEBUG [pool-1-thread-1] client.ClientScanner(224): Finished with region {ENCODED => 4d1c3ac2a4600b4f4a06552bc3422f4d, NAME => 'test,iii,1376467521631.4d1c3ac2a4600b4f4a06552bc3422f4d.', STARTKEY => 'iii', ENDKEY => 'jjj'}
2013-08-14 08:06:47,707 DEBUG [pool-1-thread-1] client.ClientScanner(232): Advancing internal scanner to startKey at 'jjj'
2013-08-14 08:06:47,708 DEBUG [pool-1-thread-1] client.ClientScanner(224): Finished with region {ENCODED => 244f51ae8d06b01e6ee90ab7c7c436e0, NAME => 'test,jjj,1376467521631.244f51ae8d06b01e6ee90ab7c7c436e0.', STARTKEY => 'jjj', ENDKEY => 'kkk'}
2013-08-14 08:06:47,708 DEBUG [pool-1-thread-1] client.ClientScanner(232): Advancing internal scanner to startKey at 'kkk'
2013-08-14 08:06:47,710 DEBUG [pool-1-thread-1] client.ClientScanner(224): Finished with region {ENCODED => b69f3a2e7380878b8c1194b56ae79d16, NAME => 'test,kkk,1376467521631.b69f3a2e7380878b8c1194b56ae79d16.', STARTKEY => 'kkk', ENDKEY => 'lll'}
2013-08-14 08:06:47,710 DEBUG [pool-1-thread-1] client.ClientScanner(232): Advancing internal scanner to startKey at 'lll'
2013-08-14 08:06:47,712 DEBUG [pool-1-thread-1] client.ClientScanner(224): Finished with region {ENCODED => 547591e713f1832c5949291c711ebea2, NAME => 'test,lll,1376467521631.547591e713f1832c5949291c711ebea2.', STARTKEY => 'lll', ENDKEY => 'mmm'}
2013-08-14 08:06:47,712 DEBUG [pool-1-thread-1] client.ClientScanner(232): Advancing internal scanner to startKey at 'mmm'
2013-08-14 08:06:47,713 DEBUG [pool-1-thread-1] client.ClientScanner(224): Finished with region {ENCODED => e53103849c25f572506f0574ec0fe00d, NAME => 'test,mmm,1376467521631.e53103849c25f572506f0574ec0fe00d.', STARTKEY => 'mmm', ENDKEY => 'nnn'}
2013-08-14 08:06:47,713 DEBUG [pool-1-thread-1] client.ClientScanner(232): Advancing internal scanner to startKey at 'nnn'
2013-08-14 08:06:47,715 DEBUG [pool-1-thread-1] client.ClientScanner(224): Finished with region {ENCODED => 464ba8e09175477a4bd3db92baa9bfc4, NAME => 'test,nnn,1376467521631.464ba8e09175477a4bd3db92baa9bfc4.', STARTKEY => 'nnn', ENDKEY => 'ooo'}
2013-08-14 08:06:47,715 DEBUG [pool-1-thread-1] client.ClientScanner(232): Advancing internal scanner to startKey at 'ooo'
2013-08-14 08:06:47,716 DEBUG [pool-1-thread-1] client.ClientScanner(224): Finished with region {ENCODED => 84b2ed68a63a2f8acbb95948e0d6bf06, NAME => 'test,ooo,1376467521631.84b2ed68a63a2f8acbb95948e0d6bf06.', STARTKEY => 'ooo', ENDKEY => 'ppp'}
2013-08-14 08:06:47,716 DEBUG [pool-1-thread-1] client.ClientScanner(232): Advancing internal scanner to startKey at 'ppp'
2013-08-14 08:06:47,717 DEBUG [pool-1-thread-1] client.ClientScanner(224): Finished with region {ENCODED => 338bd8a386d645a9a620f468af473162, NAME => 'test,ppp,1376467521631.338bd8a386d645a9a620f468af473162.', STARTKEY => 'ppp', ENDKEY => 'qqq'}
2013-08-14 08:06:47,717 DEBUG [pool-1-thread-1] client.ClientScanner(232): Advancing internal scanner to startKey at 'qqq'
2013-08-14 08:06:47,718 DEBUG [pool-1-thread-1] client.ClientScanner(224): Finished with region {ENCODED => 2a804f1993d7d476cf7d6e4a9162150c, NAME => 'test,qqq,1376467521631.2a804f1993d7d476cf7d6e4a9162150c.', STARTKEY => 'qqq', ENDKEY => 'rrr'}
2013-08-14 08:06:47,719 DEBUG [pool-1-thread-1] client.ClientScanner(232): Advancing internal scanner to startKey at 'rrr'
2013-08-14 08:06:47,720 DEBUG [pool-1-thread-1] client.ClientScanner(224): Finished with region {ENCODED => 1342f448b7753617eb641328a0ac7c58, NAME => 'test,rrr,1376467521632.1342f448b7753617eb641328a0ac7c58.', STARTKEY => 'rrr', ENDKEY => 'sss'}
2013-08-14 08:06:47,720 DEBUG [pool-1-thread-1] client.ClientScanner(232): Advancing internal scanner to startKey at 'sss'
2013-08-14 08:06:47,721 DEBUG [pool-1-thread-1] client.ClientScanner(224): Finished with region {ENCODED => 764e0f2c0ea97e372e919dfa91291147, NAME => 'test,sss,1376467521632.764e0f2c0ea97e372e919dfa91291147.', STARTKEY => 'sss', ENDKEY => 'ttt'}
2013-08-14 08:06:47,723 DEBUG [pool-1-thread-1] client.ClientScanner(232): Advancing internal scanner to startKey at 'ttt'
2013-08-14 08:06:47,724 DEBUG [pool-1-thread-1] client.ClientScanner(224): Finished with region {ENCODED => dc6662627864407718f58bf7047a0359, NAME => 'test,ttt,1376467521632.dc6662627864407718f58bf7047a0359.', STARTKEY => 'ttt', ENDKEY => 'uuu'}
2013-08-14 08:06:47,724 DEBUG [pool-1-thread-1] client.ClientScanner(232): Advancing internal scanner to startKey at 'uuu'
2013-08-14 08:06:47,725 DEBUG [pool-1-thread-1] client.ClientScanner(224): Finished with region {ENCODED => b0659e276e4cb099ffbc2b6ce1815be6, NAME => 'test,uuu,1376467521632.b0659e276e4cb099ffbc2b6ce1815be6.', STARTKEY => 'uuu', ENDKEY => 'vvv'}
2013-08-14 08:06:47,725 DEBUG [pool-1-thread-1] client.ClientScanner(232): Advancing internal scanner to startKey at 'vvv'
2013-08-14 08:06:47,726 DEBUG [pool-1-thread-1] client.ClientScanner(224): Finished with region {ENCODED => 2a215d21feab79ebd405fbe7d373c59a, NAME => 'test,vvv,1376467521632.2a215d21feab79ebd405fbe7d373c59a.', STARTKEY => 'vvv', ENDKEY => 'www'}
2013-08-14 08:06:47,727 DEBUG [pool-1-thread-1] client.ClientScanner(232): Advancing internal scanner to startKey at 'www'
2013-08-14 08:06:47,728 DEBUG [pool-1-thread-1] client.ClientScanner(224): Finished with region {ENCODED => a0a0b447554e833b5a2d655dea47d3c1, NAME => 'test,www,1376467521632.a0a0b447554e833b5a2d655dea47d3c1.', STARTKEY => 'www', ENDKEY => 'xxx'}
2013-08-14 08:06:47,728 DEBUG [pool-1-thread-1] client.ClientScanner(232): Advancing internal scanner to startKey at 'xxx'
2013-08-14 08:06:47,729 DEBUG [pool-1-thread-1] client.ClientScanner(224): Finished with region {ENCODED => f229fb895b35fd2e23739c136af89ccc, NAME => 'test,xxx,1376467521632.f229fb895b35fd2e23739c136af89ccc.', STARTKEY => 'xxx', ENDKEY => 'yyy'}
2013-08-14 08:06:47,730 DEBUG [pool-1-thread-1] client.ClientScanner(232): Advancing internal scanner to startKey at 'yyy'
2013-08-14 08:06:47,731 DEBUG [pool-1-thread-1] client.ClientScanner(224): Finished with region {ENCODED => 88921d3ada3eb162924ca7716d97f3bb, NAME => 'test,yyy,1376467521632.88921d3ada3eb162924ca7716d97f3bb.', STARTKEY => 'yyy', ENDKEY => 'zzz'}
2013-08-14 08:06:47,731 DEBUG [pool-1-thread-1] client.ClientScanner(232): Advancing internal scanner to startKey at 'zzz'
2013-08-14 08:06:47,732 DEBUG [pool-1-thread-1] client.ClientScanner(218): Finished region={ENCODED => be52648d474b32e36de1bb2cd788702e, NAME => 'test,zzz,1376467521632.be52648d474b32e36de1bb2cd788702e.', STARTKEY => 'zzz', ENDKEY => ''}
2013-08-14 08:06:47,738 DEBUG [pool-1-thread-1] client.ClientScanner(224): Finished with region {ENCODED => cc6b66f5e02b40fdb33db3a30a21cfd9, NAME => 'test,,1376467533987.cc6b66f5e02b40fdb33db3a30a21cfd9.', STARTKEY => '', ENDKEY => 'bbb'}
2013-08-14 08:06:47,738 DEBUG [pool-1-thread-1] client.ClientScanner(232): Advancing internal scanner to startKey at 'bbb'
2013-08-14 08:06:47,739 DEBUG [pool-1-thread-1] client.ClientScanner(224): Finished with region {ENCODED => b32ae9c9446aa0b6394be9246bf4e1b8, NAME => 'test,bbb,1376467533987.b32ae9c9446aa0b6394be9246bf4e1b8.', STARTKEY => 'bbb', ENDKEY => 'ccc'}
2013-08-14 08:06:47,739 DEBUG [pool-1-thread-1] client.ClientScanner(232): Advancing internal scanner to startKey at 'ccc'
2013-08-14 08:06:47,740 DEBUG [pool-1-thread-1] client.ClientScanner(224): Finished with region {ENCODED => dc08be071fc960230017af7ea91af545, NAME => 'test,ccc,1376467533987.dc08be071fc960230017af7ea91af545.', STARTKEY => 'ccc', ENDKEY => 'ddd'}
2013-08-14 08:06:47,741 DEBUG [pool-1-thread-1] client.ClientScanner(232): Advancing internal scanner to startKey at 'ddd'
2013-08-14 08:06:47,742 DEBUG [pool-1-thread-1] client.ClientScanner(224): Finished with region {ENCODED => 28622c0c2c1a2ae54cab0101d6b270ef, NAME => 'test,ddd,1376467533987.28622c0c2c1a2ae54cab0101d6b270ef.', STARTKEY => 'ddd', ENDKEY => 'eee'}
2013-08-14 08:06:47,742 DEBUG [pool-1-thread-1] client.ClientScanner(232): Advancing internal scanner to startKey at 'eee'
2013-08-14 08:06:47,743 DEBUG [pool-1-thread-1] client.ClientScanner(224): Finished with region {ENCODED => 40fc3ba1b8f2b1fd4afc3f8a79fa039a, NAME => 'test,eee,1376467533988.40fc3ba1b8f2b1fd4afc3f8a79fa039a.', STARTKEY => 'eee', ENDKEY => 'fff'}
2013-08-14 08:06:47,744 DEBUG [pool-1-thread-1] client.ClientScanner(232): Advancing internal scanner to startKey at 'fff'
2013-08-14 08:06:47,745 DEBUG [pool-1-thread-1] client.ClientScanner(224): Finished with region {ENCODED => 416e9bcb7218edf860d41eaf5f3cd0c2, NAME => 'test,fff,1376467533988.416e9bcb7218edf860d41eaf5f3cd0c2.', STARTKEY => 'fff', ENDKEY => 'ggg'}
2013-08-14 08:06:47,745 DEBUG [pool-1-thread-1] client.ClientScanner(232): Advancing internal scanner to startKey at 'ggg'
2013-08-14 08:06:47,746 DEBUG [pool-1-thread-1] client.ClientScanner(224): Finished with region {ENCODED => 129878e047b3359b5819ac1b804bd810, NAME => 'test,ggg,1376467533988.129878e047b3359b5819ac1b804bd810.', STARTKEY => 'ggg', ENDKEY => 'hhh'}
2013-08-14 08:06:47,746 DEBUG [pool-1-thread-1] client.ClientScanner(232): Advancing internal scanner to startKey at 'hhh'
2013-08-14 08:06:47,747 DEBUG [pool-1-thread-1] client.ClientScanner(224): Finished with region {ENCODED => 4921124856888e8deb687a49291df9db, NAME => 'test,hhh,1376467533988.4921124856888e8deb687a49291df9db.', STARTKEY => 'hhh', ENDKEY => 'iii'}
2013-08-14 08:06:47,748 DEBUG [pool-1-thread-1] client.ClientScanner(232): Advancing internal scanner to startKey at 'iii'
2013-08-14 08:06:47,749 DEBUG [pool-1-thread-1] client.ClientScanner(224): Finished with region {ENCODED => d4f90b4486102e6d195516d0f67af62f, NAME => 'test,iii,1376467533988.d4f90b4486102e6d195516d0f67af62f.', STARTKEY => 'iii', ENDKEY => 'jjj'}
2013-08-14 08:06:47,749 DEBUG [pool-1-thread-1] client.ClientScanner(232): Advancing internal scanner to startKey at 'jjj'
2013-08-14 08:06:47,750 DEBUG [pool-1-thread-1] client.ClientScanner(224): Finished with region {ENCODED => d8926665650acd1180a7e9e57b34e294, NAME => 'test,jjj,1376467533988.d8926665650acd1180a7e9e57b34e294.', STARTKEY => 'jjj', ENDKEY => 'kkk'}
2013-08-14 08:06:47,750 DEBUG [pool-1-thread-1] client.ClientScanner(232): Advancing internal scanner to startKey at 'kkk'
2013-08-14 08:06:47,752 DEBUG [pool-1-thread-1] client.ClientScanner(224): Finished with region {ENCODED => 2f7f4a12326becaac75a5c607704ff6e, NAME => 'test,kkk,1376467533988.2f7f4a12326becaac75a5c607704ff6e.', STARTKEY => 'kkk', ENDKEY => 'lll'}
2013-08-14 08:06:47,752 DEBUG [pool-1-thread-1] client.ClientScanner(232): Advancing internal scanner to startKey at 'lll'
2013-08-14 08:06:47,753 DEBUG [pool-1-thread-1] client.ClientScanner(224): Finished with region {ENCODED => 818ca7877f07e5769e423e1709c86f00, NAME => 'test,lll,1376467533988.818ca7877f07e5769e423e1709c86f00.', STARTKEY => 'lll', ENDKEY => 'mmm'}
2013-08-14 08:06:47,753 DEBUG [pool-1-thread-1] client.ClientScanner(232): Advancing internal scanner to startKey at 'mmm'
2013-08-14 08:06:47,755 DEBUG [pool-1-thread-1] client.ClientScanner(224): Finished with region {ENCODED => 2bd177048895ef80ebff3d1cf814c026, NAME => 'test,mmm,1376467533988.2bd177048895ef80ebff3d1cf814c026.', STARTKEY => 'mmm', ENDKEY => 'nnn'}
2013-08-14 08:06:47,755 DEBUG [pool-1-thread-1] client.ClientScanner(232): Advancing internal scanner to startKey at 'nnn'
2013-08-14 08:06:47,756 DEBUG [pool-1-thread-1] client.ClientScanner(224): Finished with region {ENCODED => 5d3e643af8044b869615e55c73906644, NAME => 'test,nnn,1376467533988.5d3e643af8044b869615e55c73906644.', STARTKEY => 'nnn', ENDKEY => 'ooo'}
2013-08-14 08:06:47,756 DEBUG [pool-1-thread-1] client.ClientScanner(232): Advancing internal scanner to startKey at 'ooo'
2013-08-14 08:06:47,757 DEBUG [pool-1-thread-1] client.ClientScanner(224): Finished with region {ENCODED => 1d0bbd585d83ddcfb1de051e76e587a4, NAME => 'test,ooo,1376467533988.1d0bbd585d83ddcfb1de051e76e587a4.', STARTKEY => 'ooo', ENDKEY => 'ppp'}
2013-08-14 08:06:47,758 DEBUG [pool-1-thread-1] client.ClientScanner(232): Advancing internal scanner to startKey at 'ppp'
2013-08-14 08:06:47,759 DEBUG [pool-1-thread-1] client.ClientScanner(224): Finished with region {ENCODED => bcad5b36210efa05976c77d4d6e042a0, NAME => 'test,ppp,1376467533988.bcad5b36210efa05976c77d4d6e042a0.', STARTKEY => 'ppp', ENDKEY => 'qqq'}
2013-08-14 08:06:47,759 DEBUG [pool-1-thread-1] client.ClientScanner(232): Advancing internal scanner to startKey at 'qqq'
2013-08-14 08:06:47,760 DEBUG [pool-1-thread-1] client.ClientScanner(224): Finished with region {ENCODED => 0a0fe7b46f5e416a424b4029985e6151, NAME => 'test,qqq,1376467533988.0a0fe7b46f5e416a424b4029985e6151.', STARTKEY => 'qqq', ENDKEY => 'rrr'}
2013-08-14 08:06:47,760 DEBUG [pool-1-thread-1] client.ClientScanner(232): Advancing internal scanner to startKey at 'rrr'
2013-08-14 08:06:47,762 DEBUG [pool-1-thread-1] client.ClientScanner(224): Finished with region {ENCODED => acce09b6cf9dfbc705a2fb13f53cf3d7, NAME => 'test,rrr,1376467533988.acce09b6cf9dfbc705a2fb13f53cf3d7.', STARTKEY => 'rrr', ENDKEY => 'sss'}
2013-08-14 08:06:47,762 DEBUG [pool-1-thread-1] client.ClientScanner(232): Advancing internal scanner to startKey at 'sss'
2013-08-14 08:06:47,763 DEBUG [pool-1-thread-1] client.ClientScanner(224): Finished with region {ENCODED => 7dfcd8e35ab1868c0299327e5c35a11c, NAME => 'test,sss,1376467533988.7dfcd8e35ab1868c0299327e5c35a11c.', STARTKEY => 'sss', ENDKEY => 'ttt'}
2013-08-14 08:06:47,763 DEBUG [pool-1-thread-1] client.ClientScanner(232): Advancing internal scanner to startKey at 'ttt'
2013-08-14 08:06:47,765 DEBUG [pool-1-thread-1] client.ClientScanner(224): Finished with region {ENCODED => 224f9abed6a534d2a1b8872a3d3d6343, NAME => 'test,ttt,1376467533988.224f9abed6a534d2a1b8872a3d3d6343.', STARTKEY => 'ttt', ENDKEY => 'uuu'}
2013-08-14 08:06:47,765 DEBUG [pool-1-thread-1] client.ClientScanner(232): Advancing internal scanner to startKey at 'uuu'
2013-08-14 08:06:47,766 DEBUG [pool-1-thread-1] client.ClientScanner(224): Finished with region {ENCODED => 851bfba81f4c6ad5fee17340bba7e01e, NAME => 'test,uuu,1376467533988.851bfba81f4c6ad5fee17340bba7e01e.', STARTKEY => 'uuu', ENDKEY => 'vvv'}
2013-08-14 08:06:47,766 DEBUG [pool-1-thread-1] client.ClientScanner(232): Advancing internal scanner to startKey at 'vvv'
2013-08-14 08:06:47,767 DEBUG [pool-1-thread-1] client.ClientScanner(224): Finished with region {ENCODED => f6c06ad2bd69f450945aaf5b969f911a, NAME => 'test,vvv,1376467533988.f6c06ad2bd69f450945aaf5b969f911a.', STARTKEY => 'vvv', ENDKEY => 'www'}
2013-08-14 08:06:47,767 DEBUG [pool-1-thread-1] client.ClientScanner(232): Advancing internal scanner to startKey at 'www'
2013-08-14 08:06:47,769 DEBUG [pool-1-thread-1] client.ClientScanner(224): Finished with region {ENCODED => cdfb3bd48199974cd03ec1fbad336cf2, NAME => 'test,www,1376467533988.cdfb3bd48199974cd03ec1fbad336cf2.', STARTKEY => 'www', ENDKEY => 'xxx'}
2013-08-14 08:06:47,769 DEBUG [pool-1-thread-1] client.ClientScanner(232): Advancing internal scanner to startKey at 'xxx'
2013-08-14 08:06:47,770 DEBUG [pool-1-thread-1] client.ClientScanner(224): Finished with region {ENCODED => 1197fb107c4c1f74d8aa76a831f8afc7, NAME => 'test,xxx,1376467533988.1197fb107c4c1f74d8aa76a831f8afc7.', STARTKEY => 'xxx', ENDKEY => 'yyy'}
2013-08-14 08:06:47,770 DEBUG [pool-1-thread-1] client.ClientScanner(232): Advancing internal scanner to startKey at 'yyy'
2013-08-14 08:06:47,772 DEBUG [pool-1-thread-1] client.ClientScanner(224): Finished with region {ENCODED => 62663d394036d30a5151411754d54e48, NAME => 'test,yyy,1376467533988.62663d394036d30a5151411754d54e48.', STARTKEY => 'yyy', ENDKEY => 'zzz'}
2013-08-14 08:06:47,772 DEBUG [pool-1-thread-1] client.ClientScanner(232): Advancing internal scanner to startKey at 'zzz'
2013-08-14 08:06:47,773 DEBUG [pool-1-thread-1] client.ClientScanner(218): Finished region={ENCODED => c092086111b214aed4557a7ad9ef963c, NAME => 'test,zzz,1376467533988.c092086111b214aed4557a7ad9ef963c.', STARTKEY => 'zzz', ENDKEY => ''}
2013-08-14 08:06:47,797 DEBUG [RS:1;vesta:39079-EventThread.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/jenkins/hbase/WALs/vesta.apache.org,39079,1376467506138/vesta.apache.org%2C39079%2C1376467506138.1376467603252 is an HLog file, so reordering blocks, last hostname will be:vesta.apache.org
2013-08-14 08:06:47,800 DEBUG [RS:1;vesta:39079-EventThread.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false
2013-08-14 08:06:47,800 DEBUG [RS:1;vesta:39079-EventThread.replicationSource,2] regionserver.ReplicationSource(578): Nothing to replicate, sleeping 100 times 1
2013-08-14 08:06:47,813 DEBUG [RS:0;vesta:43149-EventThread] zookeeper.ZooKeeperWatcher(312): regionserver:43149-0x1407bdb352a0006 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/replication/peers/2/peer-state
2013-08-14 08:06:47,813 DEBUG [RS:1;vesta:39079-EventThread] zookeeper.ZooKeeperWatcher(312): regionserver:39079-0x1407bdb352a0005 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/replication/peers/2/peer-state
2013-08-14 08:06:47,813 INFO [Thread-2823] replication.ReplicationPeersZKImpl(369): state of the peer 2 changed to DISABLED
2013-08-14 08:06:47,814 INFO [Thread-2823] replication.ReplicationPeersZKImpl(129): peer 2 is disabled
2013-08-14 08:06:47,840 INFO [Thread-2823] replication.TestReplicationSmallTests(295): Row not replicated, let's wait a bit more...
2013-08-14 08:06:47,900 DEBUG [RS:1;vesta:39079-EventThread.replicationSource,2] regionserver.ReplicationSource(578): Replication is disabled, sleeping 100 times 2
2013-08-14 08:06:48,101 DEBUG [RS:1;vesta:39079-EventThread.replicationSource,2] regionserver.ReplicationSource(578): Replication is disabled, sleeping 100 times 3
2013-08-14 08:06:48,188 DEBUG [RS:0;vesta:43149-EventThread.replicationSource,2] regionserver.ReplicationSource(578): Replication is disabled, sleeping 100 times 10
2013-08-14 08:06:48,341 INFO [Thread-2823] replication.TestReplicationSmallTests(295): Row not replicated, let's wait a bit more...
2013-08-14 08:06:48,390 INFO [vesta:43149Replication Statistics #0] regionserver.Replication$ReplicationStatisticsThread(324): Normal source for cluster 2: Total replicated edits: 2, currently replicating from: hdfs://localhost:52881/user/jenkins/hbase/WALs/vesta.apache.org,43149,1376467506109/vesta.apache.org%2C43149%2C1376467506109.1376467569820 at position: 0
2013-08-14 08:06:48,390 INFO [vesta:39079Replication Statistics #0] regionserver.Replication$ReplicationStatisticsThread(324): Normal source for cluster 2: Total replicated edits: 204, currently replicating from: hdfs://localhost:52881/user/jenkins/hbase/WALs/vesta.apache.org,39079,1376467506138/vesta.apache.org%2C39079%2C1376467506138.1376467603252 at position: 12207
2013-08-14 08:06:48,401 DEBUG [RS:1;vesta:39079-EventThread.replicationSource,2] regionserver.ReplicationSource(578): Replication is disabled, sleeping 100 times 4
2013-08-14 08:06:48,801 DEBUG [RS:1;vesta:39079-EventThread.replicationSource,2] regionserver.ReplicationSource(578): Replication is disabled, sleeping 100 times 5
2013-08-14 08:06:48,842 INFO [Thread-2823] replication.TestReplicationSmallTests(295): Row not replicated, let's wait a bit more...
2013-08-14 08:06:49,188 DEBUG [RS:0;vesta:43149-EventThread.replicationSource,2] regionserver.ReplicationSource(578): Replication is disabled, sleeping 100 times 10
2013-08-14 08:06:49,214 INFO [vesta:37982Replication Statistics #0] regionserver.Replication$ReplicationStatisticsThread(324): Sink: age in ms of last applied edit: 45427, total replicated edits: 2
2013-08-14 08:06:49,301 DEBUG [RS:1;vesta:39079-EventThread.replicationSource,2] regionserver.ReplicationSource(578): Replication is disabled, sleeping 100 times 6
2013-08-14 08:06:49,302 INFO [vesta:36402Replication Statistics #0] regionserver.Replication$ReplicationStatisticsThread(324): Sink: age in ms of last applied edit: 1616, total replicated edits: 1459
2013-08-14 08:06:49,343 INFO [Thread-2823] replication.TestReplicationSmallTests(295): Row not replicated, let's wait a bit more...
2013-08-14 08:06:49,844 INFO [Thread-2823] replication.TestReplicationSmallTests(295): Row not replicated, let's wait a bit more...
2013-08-14 08:06:49,901 DEBUG [RS:1;vesta:39079-EventThread.replicationSource,2] regionserver.ReplicationSource(578): Replication is disabled, sleeping 100 times 7
2013-08-14 08:06:50,189 DEBUG [RS:0;vesta:43149-EventThread.replicationSource,2] regionserver.ReplicationSource(578): Replication is disabled, sleeping 100 times 10
2013-08-14 08:06:50,345 INFO [Thread-2823] replication.TestReplicationSmallTests(295): Row not replicated, let's wait a bit more...
2013-08-14 08:06:50,602 DEBUG [RS:1;vesta:39079-EventThread.replicationSource,2] regionserver.ReplicationSource(578): Replication is disabled, sleeping 100 times 8
2013-08-14 08:06:50,846 INFO [Thread-2823] replication.TestReplicationSmallTests(295): Row not replicated, let's wait a bit more...
2013-08-14 08:06:51,189 DEBUG [RS:0;vesta:43149-EventThread.replicationSource,2] regionserver.ReplicationSource(578): Replication is disabled, sleeping 100 times 10
2013-08-14 08:06:51,348 INFO [Thread-2823] replication.TestReplicationSmallTests(295): Row not replicated, let's wait a bit more...
2013-08-14 08:06:51,402 DEBUG [RS:1;vesta:39079-EventThread.replicationSource,2] regionserver.ReplicationSource(578): Replication is disabled, sleeping 100 times 9
2013-08-14 08:06:51,480 DEBUG [NamespaceJanitor-vesta:59544] client.ClientScanner(218): Finished region={ENCODED => b09f7680cc8fdb352c8c8daae0548a05, NAME => 'hbase:namespace,,1376467520327.b09f7680cc8fdb352c8c8daae0548a05.', STARTKEY => '', ENDKEY => ''}
2013-08-14 08:06:51,849 INFO [Thread-2823] replication.TestReplicationSmallTests(295): Row not replicated, let's wait a bit more...
2013-08-14 08:06:52,189 DEBUG [RS:0;vesta:43149-EventThread.replicationSource,2] regionserver.ReplicationSource(578): Replication is disabled, sleeping 100 times 10
2013-08-14 08:06:52,302 DEBUG [RS:1;vesta:39079-EventThread.replicationSource,2] regionserver.ReplicationSource(578): Replication is disabled, sleeping 100 times 10
2013-08-14 08:06:52,350 INFO [Thread-2823] replication.TestReplicationSmallTests(295): Row not replicated, let's wait a bit more...
2013-08-14 08:06:53,189 DEBUG [RS:0;vesta:43149-EventThread.replicationSource,2] regionserver.ReplicationSource(578): Replication is disabled, sleeping 100 times 10
2013-08-14 08:06:53,302 DEBUG [RS:1;vesta:39079-EventThread.replicationSource,2] regionserver.ReplicationSource(578): Replication is disabled, sleeping 100 times 10
2013-08-14 08:06:53,390 DEBUG [RS:0;vesta:43149-EventThread] zookeeper.ZooKeeperWatcher(312): regionserver:43149-0x1407bdb352a0006 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/replication/peers/2/peer-state
2013-08-14 08:06:53,390 INFO [Thread-2823] replication.ReplicationPeersZKImpl(369): state of the peer 2 changed to ENABLED
2013-08-14 08:06:53,390 DEBUG [RS:1;vesta:39079-EventThread] zookeeper.ZooKeeperWatcher(312): regionserver:39079-0x1407bdb352a0005 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/replication/peers/2/peer-state
2013-08-14 08:06:53,390 INFO [vesta:39079Replication Statistics #0] regionserver.Replication$ReplicationStatisticsThread(324): Normal source for cluster 2: Total replicated edits: 204, currently replicating from: hdfs://localhost:52881/user/jenkins/hbase/WALs/vesta.apache.org,39079,1376467506138/vesta.apache.org%2C39079%2C1376467506138.1376467603252 at position: 12207
2013-08-14 08:06:53,390 INFO [vesta:43149Replication Statistics #0] regionserver.Replication$ReplicationStatisticsThread(324): Normal source for cluster 2: Total replicated edits: 2, currently replicating from: hdfs://localhost:52881/user/jenkins/hbase/WALs/vesta.apache.org,43149,1376467506109/vesta.apache.org%2C43149%2C1376467506109.1376467569820 at position: 0
2013-08-14 08:06:53,390 INFO [Thread-2823] replication.ReplicationPeersZKImpl(123): peer 2 is enabled
2013-08-14 08:06:53,392 INFO [Thread-2823] replication.TestReplicationSmallTests(306): Row not available
2013-08-14 08:06:53,893 INFO [Thread-2823] replication.TestReplicationSmallTests(306): Row not available
2013-08-14 08:06:54,191 DEBUG [RS:0;vesta:43149-EventThread.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/jenkins/hbase/WALs/vesta.apache.org,43149,1376467506109/vesta.apache.org%2C43149%2C1376467506109.1376467569820 is an HLog file, so reordering blocks, last hostname will be:vesta.apache.org
2013-08-14 08:06:54,214 INFO [vesta:37982Replication Statistics #0] regionserver.Replication$ReplicationStatisticsThread(324): Sink: age in ms of last applied edit: 50427, total replicated edits: 2
2013-08-14 08:06:54,302 INFO [vesta:36402Replication Statistics #0] regionserver.Replication$ReplicationStatisticsThread(324): Sink: age in ms of last applied edit: 6616, total replicated edits: 1459
2013-08-14 08:06:54,303 DEBUG [RS:1;vesta:39079-EventThread.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/jenkins/hbase/WALs/vesta.apache.org,39079,1376467506138/vesta.apache.org%2C39079%2C1376467506138.1376467603252 is an HLog file, so reordering blocks, last hostname will be:vesta.apache.org
2013-08-14 08:06:54,396 INFO [Thread-2823] replication.TestReplicationSmallTests(306): Row not available
2013-08-14 08:06:54,896 INFO [Thread-2823] replication.TestReplicationSmallTests(306): Row not available
2013-08-14 08:06:55,397 INFO [Thread-2823] replication.TestReplicationSmallTests(306): Row not available
2013-08-14 08:06:55,898 INFO [Thread-2823] replication.TestReplicationSmallTests(306): Row not available
2013-08-14 08:06:56,399 INFO [Thread-2823] replication.TestReplicationSmallTests(306): Row not available
2013-08-14 08:06:56,900 INFO [Thread-2823] replication.TestReplicationSmallTests(306): Row not available
2013-08-14 08:06:57,401 INFO [Thread-2823] replication.TestReplicationSmallTests(306): Row not available
2013-08-14 08:06:57,902 INFO [Thread-2823] replication.TestReplicationSmallTests(306): Row not available
2013-08-14 08:06:58,390 INFO [vesta:43149Replication Statistics #0] regionserver.Replication$ReplicationStatisticsThread(324): Normal source for cluster 2: Total replicated edits: 2, currently replicating from: hdfs://localhost:52881/user/jenkins/hbase/WALs/vesta.apache.org,43149,1376467506109/vesta.apache.org%2C43149%2C1376467506109.1376467569820 at position: 0
2013-08-14 08:06:58,390 INFO [vesta:39079Replication Statistics #0] regionserver.Replication$ReplicationStatisticsThread(324): Normal source for cluster 2: Total replicated edits: 204, currently replicating from: hdfs://localhost:52881/user/jenkins/hbase/WALs/vesta.apache.org,39079,1376467506138/vesta.apache.org%2C39079%2C1376467506138.1376467603252 at position: 12207
2013-08-14 08:06:59,861 INFO [vesta:36402Replication Statistics #0] regionserver.Replication$ReplicationStatisticsThread(324): Sink: age in ms of last applied edit: 12175, total replicated edits: 1459
2013-08-14 08:06:59,862 INFO [vesta:37982Replication Statistics #0] regionserver.Replication$ReplicationStatisticsThread(324): Sink: age in ms of last applied edit: 56075, total replicated edits: 2
2013-08-14 08:06:59,878 INFO [pool-1-thread-1] hbase.ResourceChecker(171): after: replication.TestReplicationSmallTests#testDisableEnable Thread=493 (was 597), OpenFileDescriptor=715 (was 893), MaxFileDescriptor=4096 (was 4096), SystemLoadAverage=857 (was 792) - SystemLoadAverage LEAK? -, ProcessCount=135 (was 135), AvailableMemoryMB=813 (was 442) - AvailableMemoryMB LEAK? -, ConnectionCount=9 (was 9)
|