2013-05-24 22:00:26,256 INFO [pool-1-thread-1] hbase.ResourceChecker(147): before: replication.TestReplicationQueueFailoverCompressed#queueFailover Thread=513, OpenFileDescriptor=701, MaxFileDescriptor=32768, SystemLoadAverage=835, ProcessCount=483, AvailableMemoryMB=6810, ConnectionCount=8 2013-05-24 22:00:26,257 WARN [pool-1-thread-1] hbase.ResourceChecker(134): Thread=513 is superior to 500 2013-05-24 22:00:26,259 INFO [Thread-584] replication.TestReplicationQueueFailover(61): Start loading table 2013-05-24 22:00:26,827 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432809393 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:26,828 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432809419 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:26,831 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432809393 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:26,831 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(583): Unable to open a reader, sleeping 100 times 10 2013-05-24 22:00:26,870 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:26,940 INFO [REPL IPC Server handler 0 on 51523] zookeeper.RecoverableZooKeeper(119): Process identifier=hconnection-0x4b76ffeb connecting to ZooKeeper ensemble=localhost:51411 2013-05-24 22:00:26,974 DEBUG [REPL IPC Server handler 0 on 51523-EventThread] zookeeper.ZooKeeperWatcher(307): hconnection-0x4b76ffeb Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2013-05-24 22:00:26,975 DEBUG [REPL IPC Server handler 0 on 51523-EventThread] zookeeper.ZooKeeperWatcher(384): hconnection-0x4b76ffeb-0x13ed88e2cdf001a connected 2013-05-24 22:00:26,985 DEBUG [REPL IPC Server handler 0 on 51523] client.MetaScanner(160): Scanning .META. starting at row=test,,00000000000000 for max=10 with caching=10 2013-05-24 22:00:26,985 DEBUG [REPL IPC Server handler 0 on 51523] client.ClientScanner(96): Creating scanner over .META. starting at key 'test,,00000000000000' 2013-05-24 22:00:26,986 DEBUG [REPL IPC Server handler 0 on 51523] client.ClientScanner(209): Advancing internal scanner to startKey at 'test,,00000000000000' 2013-05-24 22:00:26,991 DEBUG [REPL IPC Server handler 0 on 51523] client.ClientScanner(196): Finished scanning region {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,} 2013-05-24 22:00:26,991 DEBUG [REPL IPC Server handler 0 on 51523] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,,1369432825000.45883ac87958f956040ff389c9f7eb86. is kiyo.gq1.ygridcore.net:39146 2013-05-24 22:00:26,991 INFO [REPL IPC Server handler 0 on 51523] client.HConnectionManager$HConnectionImplementation(1640): Closing zookeeper sessionid=0x13ed88e2cdf001a 2013-05-24 22:00:27,246 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432809419 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:27,252 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:27,417 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432809419 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:27,427 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:27,448 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] regionserver.ReplicationSource(583): Nothing to replicate, sleeping 100 times 1 2013-05-24 22:00:27,550 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432809419 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:27,556 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:27,574 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] regionserver.ReplicationSource(583): Nothing to replicate, sleeping 100 times 2 2013-05-24 22:00:27,776 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432809419 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:27,781 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:27,796 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] regionserver.ReplicationSource(583): Nothing to replicate, sleeping 100 times 3 2013-05-24 22:00:27,833 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432809393 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:27,838 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:27,893 INFO [REPL IPC Server handler 0 on 39146] zookeeper.RecoverableZooKeeper(119): Process identifier=hconnection-0x2d437a6d connecting to ZooKeeper ensemble=localhost:51411 2013-05-24 22:00:28,011 DEBUG [REPL IPC Server handler 0 on 39146-EventThread] zookeeper.ZooKeeperWatcher(307): hconnection-0x2d437a6d Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2013-05-24 22:00:28,013 DEBUG [REPL IPC Server handler 0 on 39146-EventThread] zookeeper.ZooKeeperWatcher(384): hconnection-0x2d437a6d-0x13ed88e2cdf001b connected 2013-05-24 22:00:28,018 DEBUG [REPL IPC Server handler 0 on 39146] client.MetaScanner(160): Scanning .META. starting at row=test,,00000000000000 for max=10 with caching=10 2013-05-24 22:00:28,019 DEBUG [REPL IPC Server handler 0 on 39146] client.ClientScanner(96): Creating scanner over .META. starting at key 'test,,00000000000000' 2013-05-24 22:00:28,019 DEBUG [REPL IPC Server handler 0 on 39146] client.ClientScanner(209): Advancing internal scanner to startKey at 'test,,00000000000000' 2013-05-24 22:00:28,024 DEBUG [REPL IPC Server handler 0 on 39146] client.ClientScanner(196): Finished scanning region {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,} 2013-05-24 22:00:28,024 DEBUG [REPL IPC Server handler 0 on 39146] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,,1369432825000.45883ac87958f956040ff389c9f7eb86. is kiyo.gq1.ygridcore.net:39146 2013-05-24 22:00:28,024 INFO [REPL IPC Server handler 0 on 39146] client.HConnectionManager$HConnectionImplementation(1640): Closing zookeeper sessionid=0x13ed88e2cdf001b 2013-05-24 22:00:28,098 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432809419 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:28,106 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:28,120 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] regionserver.ReplicationSource(583): Nothing to replicate, sleeping 100 times 4 2013-05-24 22:00:28,177 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.logRoller] regionserver.LogRoller(90): HLog roll requested 2013-05-24 22:00:28,224 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432809393 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:28,230 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:28,298 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.logRoller] wal.FSHLog(658): cleanupCurrentWriter waiting for transactions to get synced total 469 synced till here 468 2013-05-24 22:00:28,302 INFO [IPC Server handler 9 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53333 is added to blk_2355220913760586106_1035{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:59993|RBW], ReplicaUnderConstruction[127.0.0.1:53333|RBW]]} size 0 2013-05-24 22:00:28,303 INFO [IPC Server handler 8 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59993 is added to blk_2355220913760586106_1035 size 45828 2013-05-24 22:00:28,319 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,39146,1369432814418.logRoller] regionserver.LogRoller(90): HLog roll requested 2013-05-24 22:00:28,342 INFO [kiyo.gq1.ygridcore.net,39146,1369432814418Replication Statistics #0] regionserver.Replication$ReplicationStatisticsThread(291): Sink: age in ms of last applied edit: 117, total replicated edits: 436 2013-05-24 22:00:28,343 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.logRoller] wal.FSHLog(522): Rolled WAL /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432809393, entries=469, filesize=44.8 K; new WAL=/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432828177 2013-05-24 22:00:28,376 INFO [IPC Server handler 5 on 54104] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33054 is added to blk_4718170699848835460_1009{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:52910|RBW], ReplicaUnderConstruction[127.0.0.1:33054|RBW]]} size 0 2013-05-24 22:00:28,376 INFO [IPC Server handler 7 on 54104] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52910 is added to blk_4718170699848835460_1009 size 46425 2013-05-24 22:00:28,378 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432809393 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:28,382 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 45820, fileLength: 45828, trailerPresent: true 2013-05-24 22:00:28,399 INFO [kiyo.gq1.ygridcore.net,51523,1369432814392Replication Statistics #0] regionserver.Replication$ReplicationStatisticsThread(291): Sink: age in ms of last applied edit: 1332, total replicated edits: 235 2013-05-24 22:00:28,412 INFO [RegionServer:1;kiyo.gq1.ygridcore.net,39146,1369432814418.logRoller] wal.FSHLog(522): Rolled WAL /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,39146,1369432814418/kiyo.gq1.ygridcore.net%2C39146%2C1369432814418.1369432818197, entries=4, filesize=45.3 K; new WAL=/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,39146,1369432814418/kiyo.gq1.ygridcore.net%2C39146%2C1369432814418.1369432828321 2013-05-24 22:00:28,458 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432809393 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:28,461 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 45820, fileLength: 45828, trailerPresent: true 2013-05-24 22:00:28,487 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432828177 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:28,493 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:28,522 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432809419 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:28,530 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:28,533 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432828177 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:28,542 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:28,552 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] regionserver.ReplicationSource(583): Nothing to replicate, sleeping 100 times 5 2013-05-24 22:00:28,572 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432828177 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:28,576 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:28,599 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432828177 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:28,604 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:28,630 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432828177 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:28,635 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:28,663 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432828177 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:28,669 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:28,699 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432828177 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:28,706 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:28,736 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432828177 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:28,742 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:28,772 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432828177 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:28,778 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:28,807 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432828177 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:28,813 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:28,840 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432828177 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:28,847 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:28,874 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432828177 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:28,880 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:29,054 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432809419 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:29,061 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:29,065 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432828177 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:29,071 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:29,072 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] regionserver.ReplicationSource(583): Nothing to replicate, sleeping 100 times 6 2013-05-24 22:00:29,099 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432828177 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:29,105 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:29,161 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432828177 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:29,167 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:29,288 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432828177 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:29,293 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:29,332 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432828177 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:29,337 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:29,409 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.logRoller] regionserver.LogRoller(90): HLog roll requested 2013-05-24 22:00:29,463 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432828177 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:29,469 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:29,516 INFO [IPC Server handler 3 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53333 is added to blk_-8480512510701967604_1036{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:59993|RBW], ReplicaUnderConstruction[127.0.0.1:53333|RBW]]} size 0 2013-05-24 22:00:29,517 INFO [IPC Server handler 5 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59993 is added to blk_-8480512510701967604_1036 size 46182 2013-05-24 22:00:29,540 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.logRoller] wal.FSHLog(522): Rolled WAL /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432828177, entries=471, filesize=45.1 K; new WAL=/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432829410 2013-05-24 22:00:29,540 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432828177 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:29,543 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 46174, fileLength: 46182, trailerPresent: true 2013-05-24 22:00:29,576 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432828177 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:29,579 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 46174, fileLength: 46182, trailerPresent: true 2013-05-24 22:00:29,590 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432829410 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:29,594 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432829410 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:29,595 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(583): Unable to open a reader, sleeping 100 times 1 2013-05-24 22:00:29,674 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432809419 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:29,696 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432829410 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:29,699 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432829410 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:29,700 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(583): Unable to open a reader, sleeping 100 times 2 2013-05-24 22:00:29,723 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:29,746 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,39146,1369432814418.logRoller] regionserver.LogRoller(90): HLog roll requested 2013-05-24 22:00:29,791 INFO [kiyo.gq1.ygridcore.net,49701,1369432806997Replication Statistics #0] regionserver.Replication$ReplicationStatisticsThread(291): Normal source for cluster 2: Total replicated edits: 235, currently replicating from: hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432809419 at position: N/A 2013-05-24 22:00:29,791 INFO [kiyo.gq1.ygridcore.net,42690,1369432806911Replication Statistics #0] regionserver.Replication$ReplicationStatisticsThread(291): Normal source for cluster 2: Total replicated edits: 940, currently replicating from: hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432829410 at position: N/A 2013-05-24 22:00:29,855 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.logRoller] regionserver.LogRoller(90): HLog roll requested 2013-05-24 22:00:29,902 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432829410 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:29,905 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432829410 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:29,905 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(583): Unable to open a reader, sleeping 100 times 3 2013-05-24 22:00:30,207 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432829410 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:30,210 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432829410 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:30,211 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(583): Unable to open a reader, sleeping 100 times 4 2013-05-24 22:00:30,235 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432809419 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:30,259 INFO [IPC Server handler 7 on 54104] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33054 is added to blk_5979311313555125270_1010{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:52910|RBW], ReplicaUnderConstruction[127.0.0.1:33054|RBW]]} size 0 2013-05-24 22:00:30,260 INFO [IPC Server handler 6 on 54104] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52910 is added to blk_5979311313555125270_1010 size 41030 2013-05-24 22:00:30,288 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:30,302 INFO [RegionServer:1;kiyo.gq1.ygridcore.net,39146,1369432814418.logRoller] wal.FSHLog(522): Rolled WAL /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,39146,1369432814418/kiyo.gq1.ygridcore.net%2C39146%2C1369432814418.1369432828321, entries=21, filesize=40.1 K; new WAL=/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,39146,1369432814418/kiyo.gq1.ygridcore.net%2C39146%2C1369432814418.1369432829746 2013-05-24 22:00:30,337 INFO [IPC Server handler 5 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53333 is added to blk_-4727786277570181235_1034{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:59993|RBW], ReplicaUnderConstruction[127.0.0.1:53333|RBW]]} size 68792 2013-05-24 22:00:30,338 INFO [IPC Server handler 0 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59993 is added to blk_-4727786277570181235_1034 size 68792 2013-05-24 22:00:30,612 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432829410 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:30,615 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432829410 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:30,615 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(583): Unable to open a reader, sleeping 100 times 5 2013-05-24 22:00:30,779 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432809419 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:30,783 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 68784, fileLength: 68792, trailerPresent: true 2013-05-24 22:00:30,815 INFO [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.logRoller] wal.FSHLog(522): Rolled WAL /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432809419, entries=703, filesize=67.2 K; new WAL=/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432829856 2013-05-24 22:00:30,815 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432809419 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:30,823 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 68784, fileLength: 68792, trailerPresent: true 2013-05-24 22:00:30,834 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] regionserver.ReplicationSource(583): Nothing to replicate, sleeping 100 times 1 2013-05-24 22:00:30,936 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432809419 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:30,938 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 68784, fileLength: 68792, trailerPresent: true 2013-05-24 22:00:30,947 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432829856 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:30,954 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:31,118 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432829410 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:31,123 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:31,139 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432829856 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:31,142 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:31,144 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] regionserver.ReplicationSource(583): Nothing to replicate, sleeping 100 times 1 2013-05-24 22:00:31,177 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432829410 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:31,181 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:31,203 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432829410 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:31,207 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:31,246 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432829856 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:31,249 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:31,251 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] regionserver.ReplicationSource(583): Nothing to replicate, sleeping 100 times 2 2013-05-24 22:00:31,259 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432829410 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:31,264 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:31,453 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432829856 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:31,458 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:31,460 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] regionserver.ReplicationSource(583): Nothing to replicate, sleeping 100 times 3 2013-05-24 22:00:31,573 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432829410 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:31,577 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:31,597 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,39146,1369432814418.logRoller] regionserver.LogRoller(90): HLog roll requested 2013-05-24 22:00:31,684 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432829410 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:31,688 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:31,695 INFO [IPC Server handler 8 on 54104] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33054 is added to blk_-4843105969230857017_1011{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:52910|RBW], ReplicaUnderConstruction[127.0.0.1:33054|RBW]]} size 0 2013-05-24 22:00:31,696 INFO [IPC Server handler 6 on 54104] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52910 is added to blk_-4843105969230857017_1011 size 43873 2013-05-24 22:00:31,749 INFO [RegionServer:1;kiyo.gq1.ygridcore.net,39146,1369432814418.logRoller] wal.FSHLog(522): Rolled WAL /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,39146,1369432814418/kiyo.gq1.ygridcore.net%2C39146%2C1369432814418.1369432829746, entries=8, filesize=42.8 K; new WAL=/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,39146,1369432814418/kiyo.gq1.ygridcore.net%2C39146%2C1369432814418.1369432831597 2013-05-24 22:00:31,762 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432829856 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:31,766 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:31,768 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] regionserver.ReplicationSource(583): Nothing to replicate, sleeping 100 times 4 2013-05-24 22:00:31,782 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.logRoller] regionserver.LogRoller(90): HLog roll requested 2013-05-24 22:00:31,848 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432829410 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:31,848 INFO [IPC Server handler 7 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59993 is added to blk_-8899939895098479338_1038{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:53333|RBW], ReplicaUnderConstruction[127.0.0.1:59993|RBW]]} size 0 2013-05-24 22:00:31,849 INFO [IPC Server handler 8 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53333 is added to blk_-8899939895098479338_1038 size 46052 2013-05-24 22:00:31,870 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 46044, fileLength: 46052, trailerPresent: true 2013-05-24 22:00:31,871 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.logRoller] wal.FSHLog(522): Rolled WAL /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432829410, entries=470, filesize=45.0 K; new WAL=/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432831783 2013-05-24 22:00:31,941 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432829410 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:31,943 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 46044, fileLength: 46052, trailerPresent: true 2013-05-24 22:00:31,956 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432831783 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:31,959 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432831783 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:31,959 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(583): Unable to open a reader, sleeping 100 times 1 2013-05-24 22:00:32,061 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432831783 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:32,063 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432831783 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:32,064 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(583): Unable to open a reader, sleeping 100 times 2 2013-05-24 22:00:32,104 DEBUG [Thread-584] client.MetaScanner(160): Scanning .META. starting at row=test,kkk,00000000000000 for max=10 with caching=10 2013-05-24 22:00:32,105 DEBUG [Thread-584] client.ClientScanner(96): Creating scanner over .META. starting at key 'test,kkk,00000000000000' 2013-05-24 22:00:32,105 DEBUG [Thread-584] client.ClientScanner(209): Advancing internal scanner to startKey at 'test,kkk,00000000000000' 2013-05-24 22:00:32,111 DEBUG [Thread-584] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,kkk,1369432820320.1ce9aafaaa422cef4ce0e1895111c733. is kiyo.gq1.ygridcore.net:49701 2013-05-24 22:00:32,111 DEBUG [Thread-584] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,lll,1369432820320.307a0833b0558971bccf01de5c996ebc. is kiyo.gq1.ygridcore.net:42690 2013-05-24 22:00:32,111 DEBUG [Thread-584] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,mmm,1369432820320.ab7b9b9773243cff0ac09f3c379e33dc. is kiyo.gq1.ygridcore.net:49701 2013-05-24 22:00:32,112 DEBUG [Thread-584] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,nnn,1369432820320.53e3aef4cc8815dd31923659bff5ca21. is kiyo.gq1.ygridcore.net:49701 2013-05-24 22:00:32,112 DEBUG [Thread-584] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,ooo,1369432820320.0d6df81a73ad3b2a79add80716da62ac. is kiyo.gq1.ygridcore.net:49701 2013-05-24 22:00:32,112 DEBUG [Thread-584] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,ppp,1369432820320.da89687648e34409286fd02380093aed. is kiyo.gq1.ygridcore.net:42690 2013-05-24 22:00:32,112 DEBUG [Thread-584] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,qqq,1369432820320.ec0b9e20bb8d4dc8fee9b3e80c5cae32. is kiyo.gq1.ygridcore.net:49701 2013-05-24 22:00:32,113 DEBUG [Thread-584] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,rrr,1369432820320.85815417b0e1742517fac3960fac86c1. is kiyo.gq1.ygridcore.net:42690 2013-05-24 22:00:32,113 DEBUG [Thread-584] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,sss,1369432820320.872960a61a4ceeb9f64577d8b7cb0ddf. is kiyo.gq1.ygridcore.net:49701 2013-05-24 22:00:32,113 DEBUG [Thread-584] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,ttt,1369432820320.f53d942877419dd243e3fb7e46c2cf97. is kiyo.gq1.ygridcore.net:42690 2013-05-24 22:00:32,170 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432829856 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:32,175 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:32,229 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432829856 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:32,234 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:32,264 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432829856 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:32,265 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432831783 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:32,270 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432831783 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:32,270 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(583): Unable to open a reader, sleeping 100 times 3 2013-05-24 22:00:32,271 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:32,288 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,39146,1369432814418.logRoller] regionserver.LogRoller(90): HLog roll requested 2013-05-24 22:00:32,304 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432829856 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:32,309 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:32,341 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.logRoller] regionserver.LogRoller(90): HLog roll requested 2013-05-24 22:00:32,345 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432829856 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:32,356 INFO [IPC Server handler 6 on 54104] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52910 is added to blk_-1246090825901166751_1012{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:33054|RBW], ReplicaUnderConstruction[127.0.0.1:52910|RBW]]} size 0 2013-05-24 22:00:32,356 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:32,357 INFO [IPC Server handler 5 on 54104] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33054 is added to blk_-1246090825901166751_1012 size 42319 2013-05-24 22:00:32,388 INFO [RegionServer:1;kiyo.gq1.ygridcore.net,39146,1369432814418.logRoller] wal.FSHLog(522): Rolled WAL /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,39146,1369432814418/kiyo.gq1.ygridcore.net%2C39146%2C1369432814418.1369432831597, entries=6, filesize=41.3 K; new WAL=/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,39146,1369432814418/kiyo.gq1.ygridcore.net%2C39146%2C1369432814418.1369432832288 2013-05-24 22:00:32,544 INFO [IPC Server handler 0 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59993 is added to blk_-1799298560960438661_1037{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:53333|RBW], ReplicaUnderConstruction[127.0.0.1:59993|RBW]]} size 0 2013-05-24 22:00:32,546 INFO [IPC Server handler 2 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53333 is added to blk_-1799298560960438661_1037 size 46234 2013-05-24 22:00:32,572 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432831783 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:32,577 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:32,647 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432829856 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:32,650 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 46226, fileLength: 46234, trailerPresent: true 2013-05-24 22:00:32,677 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432831783 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:32,681 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:32,691 INFO [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.logRoller] wal.FSHLog(522): Rolled WAL /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432829856, entries=472, filesize=45.2 K; new WAL=/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432832342 2013-05-24 22:00:32,698 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432829856 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:32,701 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 46226, fileLength: 46234, trailerPresent: true 2013-05-24 22:00:32,710 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432832342 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:32,711 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432831783 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:32,713 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432832342 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:32,713 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] regionserver.ReplicationSource(583): Unable to open a reader, sleeping 100 times 1 2013-05-24 22:00:32,716 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:32,751 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432831783 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:32,755 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:32,770 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432831783 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:32,774 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:32,797 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432831783 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:32,801 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:32,815 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432832342 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:32,817 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432832342 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:32,817 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] regionserver.ReplicationSource(583): Unable to open a reader, sleeping 100 times 2 2013-05-24 22:00:32,822 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432831783 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:32,827 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:32,860 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432831783 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:32,864 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:32,991 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432831783 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:32,995 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:33,075 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432832342 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:33,079 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:33,180 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432831783 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:33,184 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:33,187 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(583): Nothing to replicate, sleeping 100 times 1 2013-05-24 22:00:33,226 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432832342 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:33,231 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:33,290 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432831783 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:33,293 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:33,296 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(583): Nothing to replicate, sleeping 100 times 2 2013-05-24 22:00:33,337 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432832342 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:33,342 INFO [kiyo.gq1.ygridcore.net,39146,1369432814418Replication Statistics #0] regionserver.Replication$ReplicationStatisticsThread(291): Sink: age in ms of last applied edit: 442, total replicated edits: 1645 2013-05-24 22:00:33,344 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:33,356 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,39146,1369432814418.logRoller] regionserver.LogRoller(90): HLog roll requested 2013-05-24 22:00:33,399 INFO [kiyo.gq1.ygridcore.net,51523,1369432814392Replication Statistics #0] regionserver.Replication$ReplicationStatisticsThread(291): Sink: age in ms of last applied edit: 61, total replicated edits: 1421 2013-05-24 22:00:33,413 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432832342 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:33,417 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:33,493 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432832342 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:33,498 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432831783 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:33,502 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:33,505 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(583): Nothing to replicate, sleeping 100 times 3 2013-05-24 22:00:33,520 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:33,548 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.logRoller] regionserver.LogRoller(90): HLog roll requested 2013-05-24 22:00:33,601 INFO [IPC Server handler 8 on 54104] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33054 is added to blk_4678086675445324849_1013{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:52910|RBW], ReplicaUnderConstruction[127.0.0.1:33054|RBW]]} size 0 2013-05-24 22:00:33,602 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432832342 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:33,602 INFO [IPC Server handler 5 on 54104] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52910 is added to blk_4678086675445324849_1013 size 49811 2013-05-24 22:00:33,606 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:33,762 INFO [Thread-585-EventThread] hbase.HBaseTestingUtility$1(1837): Monitor ZKW received event=WatchedEvent state:SyncConnected type:None path:null 2013-05-24 22:00:33,808 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432831783 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:33,811 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:33,814 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(583): Nothing to replicate, sleeping 100 times 4 2013-05-24 22:00:33,857 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:42690-0x13ed88e2cdf0006 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/rs/kiyo.gq1.ygridcore.net,49701,1369432806997 2013-05-24 22:00:33,858 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/rs/kiyo.gq1.ygridcore.net,49701,1369432806997 2013-05-24 22:00:33,858 INFO [pool-1-thread-1-EventThread] zookeeper.RegionServerTracker(94): RegionServer ephemeral node deleted, processing expiration [kiyo.gq1.ygridcore.net,49701,1369432806997] 2013-05-24 22:00:33,858 INFO [Thread-585] hbase.HBaseTestingUtility(1853): ZK Closed Session 0x13ed88e2cdf0005 2013-05-24 22:00:33,862 DEBUG [pool-1-thread-1-EventThread] master.AssignmentManager(2991): based on AM, current region=.META.,,1.1028785192 is on server=kiyo.gq1.ygridcore.net,42690,1369432806911 server being checked: kiyo.gq1.ygridcore.net,49701,1369432806997 2013-05-24 22:00:33,862 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:49701-0x13ed88e2cdf0005 Received ZooKeeper Event, type=None, state=Disconnected, path=null 2013-05-24 22:00:33,862 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997-EventThread] zookeeper.ZooKeeperWatcher(389): regionserver:49701-0x13ed88e2cdf0005 Received Disconnected from ZooKeeper, ignoring 2013-05-24 22:00:33,863 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.ZKUtil(431): regionserver:42690-0x13ed88e2cdf0006 Set watcher on existing znode=/1/rs/kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:33,863 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] regionserver.ReplicationSourceManager$OtherRegionServerWatcher(467): /1/rs/kiyo.gq1.ygridcore.net,49701,1369432806997 znode expired, trying to lock it 2013-05-24 22:00:33,863 DEBUG [pool-1-thread-1-EventThread] master.ServerManager(511): Added=kiyo.gq1.ygridcore.net,49701,1369432806997 to dead servers, submitted shutdown handler to be executed meta=false 2013-05-24 22:00:33,863 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/rs 2013-05-24 22:00:33,864 INFO [Thread-585-EventThread] hbase.HBaseTestingUtility$1(1837): Monitor ZKW received event=WatchedEvent state:Disconnected type:None path:null 2013-05-24 22:00:33,866 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:42690-0x13ed88e2cdf0006 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/rs 2013-05-24 22:00:33,868 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(431): master:49101-0x13ed88e2cdf0004 Set watcher on existing znode=/1/rs/kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:33,869 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.ZKUtil(431): regionserver:42690-0x13ed88e2cdf0006 Set watcher on existing znode=/1/rs/kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:33,870 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] client.ClientScanner(96): Creating scanner over .META. starting at key '' 2013-05-24 22:00:33,870 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] client.ClientScanner(209): Advancing internal scanner to startKey at '' 2013-05-24 22:00:33,877 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] client.ClientScanner(196): Finished scanning region {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,} 2013-05-24 22:00:33,881 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] handler.ServerShutdownHandler(184): Splitting logs for kiyo.gq1.ygridcore.net,49701,1369432806997 before assignment. 2013-05-24 22:00:33,885 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] client.ClientScanner(96): Creating scanner over .META. starting at key '' 2013-05-24 22:00:33,885 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] client.ClientScanner(209): Advancing internal scanner to startKey at '' 2013-05-24 22:00:33,894 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] client.ClientScanner(196): Finished scanning region {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,} 2013-05-24 22:00:33,899 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] zookeeper.ZKUtil(690): master:49101-0x13ed88e2cdf0004 Unable to get data of znode /1/recovering-regions/a88e8c9e73f832adb3635d6996d9e72d because node does not exist (not an error) 2013-05-24 22:00:34,011 INFO [IPC Server handler 4 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53333 is added to blk_-8921468500416752551_1040{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:59993|RBW], ReplicaUnderConstruction[127.0.0.1:53333|RBW]]} size 0 2013-05-24 22:00:34,034 INFO [RegionServer:1;kiyo.gq1.ygridcore.net,39146,1369432814418.logRoller] wal.FSHLog(522): Rolled WAL /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,39146,1369432814418/kiyo.gq1.ygridcore.net%2C39146%2C1369432814418.1369432832288, entries=16, filesize=48.6 K; new WAL=/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,39146,1369432814418/kiyo.gq1.ygridcore.net%2C39146%2C1369432814418.1369432833356 2013-05-24 22:00:34,034 INFO [IPC Server handler 9 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59993 is added to blk_-8921468500416752551_1040 size 69068 2013-05-24 22:00:34,051 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.SplitLogManager(1117): Mark region a88e8c9e73f832adb3635d6996d9e72d recovering from failed region server kiyo.gq1.ygridcore.net,49701,1369432806997 2013-05-24 22:00:34,056 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] zookeeper.ZKUtil(690): master:49101-0x13ed88e2cdf0004 Unable to get data of znode /1/recovering-regions/f852f52d87b0b7acd213f5b49074dde6 because node does not exist (not an error) 2013-05-24 22:00:34,146 INFO [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.logRoller] wal.FSHLog(522): Rolled WAL /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432832342, entries=705, filesize=67.4 K; new WAL=/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432833549 2013-05-24 22:00:34,164 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.SplitLogManager(1117): Mark region f852f52d87b0b7acd213f5b49074dde6 recovering from failed region server kiyo.gq1.ygridcore.net,49701,1369432806997 2013-05-24 22:00:34,165 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] zookeeper.ZKUtil(690): master:49101-0x13ed88e2cdf0004 Unable to get data of znode /1/recovering-regions/4b3ff3efde12c5afe1220e81d155ac73 because node does not exist (not an error) 2013-05-24 22:00:34,215 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.logRoller] regionserver.LogRoller(90): HLog roll requested 2013-05-24 22:00:34,216 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432831783 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:34,221 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:34,322 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.SplitLogManager(1117): Mark region 4b3ff3efde12c5afe1220e81d155ac73 recovering from failed region server kiyo.gq1.ygridcore.net,49701,1369432806997 2013-05-24 22:00:34,371 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] zookeeper.ZKUtil(690): master:49101-0x13ed88e2cdf0004 Unable to get data of znode /1/recovering-regions/eb38c552d2a930aa1ce862202da80908 because node does not exist (not an error) 2013-05-24 22:00:34,374 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432831783 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:34,414 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:34,673 INFO [IPC Server handler 7 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53333 is added to blk_210413026503171704_1039{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:59993|RBW], ReplicaUnderConstruction[127.0.0.1:53333|RBW]]} size 0 2013-05-24 22:00:34,674 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432831783 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:34,674 INFO [IPC Server handler 3 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59993 is added to blk_210413026503171704_1039 size 46052 2013-05-24 22:00:34,714 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 46044, fileLength: 46052, trailerPresent: true 2013-05-24 22:00:34,716 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.SplitLogManager(1117): Mark region eb38c552d2a930aa1ce862202da80908 recovering from failed region server kiyo.gq1.ygridcore.net,49701,1369432806997 2013-05-24 22:00:34,717 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] zookeeper.ZKUtil(690): master:49101-0x13ed88e2cdf0004 Unable to get data of znode /1/recovering-regions/1ce9aafaaa422cef4ce0e1895111c733 because node does not exist (not an error) 2013-05-24 22:00:34,720 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(583): Nothing to replicate, sleeping 100 times 1 2013-05-24 22:00:34,756 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.SplitLogManager(1117): Mark region 1ce9aafaaa422cef4ce0e1895111c733 recovering from failed region server kiyo.gq1.ygridcore.net,49701,1369432806997 2013-05-24 22:00:34,756 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.logRoller] wal.FSHLog(522): Rolled WAL /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432831783, entries=470, filesize=45.0 K; new WAL=/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432834215 2013-05-24 22:00:34,757 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] zookeeper.ZKUtil(690): master:49101-0x13ed88e2cdf0004 Unable to get data of znode /1/recovering-regions/ab7b9b9773243cff0ac09f3c379e33dc because node does not exist (not an error) 2013-05-24 22:00:34,791 INFO [kiyo.gq1.ygridcore.net,42690,1369432806911Replication Statistics #0] regionserver.Replication$ReplicationStatisticsThread(291): Normal source for cluster 2: Total replicated edits: 1880, currently replicating from: hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432831783 at position: N/A 2013-05-24 22:00:34,791 INFO [kiyo.gq1.ygridcore.net,49701,1369432806997Replication Statistics #0] regionserver.Replication$ReplicationStatisticsThread(291): Normal source for cluster 2: Total replicated edits: 1557, currently replicating from: hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432832342 at position: N/A 2013-05-24 22:00:34,822 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432831783 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:34,824 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 46044, fileLength: 46052, trailerPresent: true 2013-05-24 22:00:34,832 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432834215 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:34,835 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.SplitLogManager(1117): Mark region ab7b9b9773243cff0ac09f3c379e33dc recovering from failed region server kiyo.gq1.ygridcore.net,49701,1369432806997 2013-05-24 22:00:34,836 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] zookeeper.ZKUtil(690): master:49101-0x13ed88e2cdf0004 Unable to get data of znode /1/recovering-regions/53e3aef4cc8815dd31923659bff5ca21 because node does not exist (not an error) 2013-05-24 22:00:34,837 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:34,859 DEBUG [IPC Server handler 4 on 49101] master.ServerManager(337): Server REPORT rejected; currently processing kiyo.gq1.ygridcore.net,49701,1369432806997 as dead server 2013-05-24 22:00:34,862 FATAL [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.HRegionServer(1705): ABORTING region server kiyo.gq1.ygridcore.net,49701,1369432806997: org.apache.hadoop.hbase.exceptions.YouAreDeadException: Server REPORT rejected; currently processing kiyo.gq1.ygridcore.net,49701,1369432806997 as dead server at org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:338) at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:253) at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:1227) at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:3800) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2103) at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1810) org.apache.hadoop.hbase.exceptions.YouAreDeadException: org.apache.hadoop.hbase.exceptions.YouAreDeadException: Server REPORT rejected; currently processing kiyo.gq1.ygridcore.net,49701,1369432806997 as dead server at org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:338) at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:253) at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:1227) at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:3800) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2103) at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1810) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:227) at org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:974) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:821) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:158) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:110) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:142) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:337) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1457) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.util.Methods.call(Methods.java:41) at org.apache.hadoop.hbase.security.User.call(User.java:420) at org.apache.hadoop.hbase.security.User.access$300(User.java:51) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:260) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:140) at java.lang.Thread.run(Thread.java:662) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.exceptions.YouAreDeadException): org.apache.hadoop.hbase.exceptions.YouAreDeadException: Server REPORT rejected; currently processing kiyo.gq1.ygridcore.net,49701,1369432806997 as dead server at org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:338) at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:253) at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:1227) at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:3800) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2103) at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1810) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1335) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1539) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1596) at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerReport(RegionServerStatusProtos.java:4095) at org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:972) ... 17 more 2013-05-24 22:00:34,863 FATAL [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.HRegionServer(1713): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2013-05-24 22:00:34,865 ERROR [IPC Server handler 1 on 49101] master.HMaster(1246): Region server kiyo.gq1.ygridcore.net,49701,1369432806997 reported a fatal error: ABORTING region server kiyo.gq1.ygridcore.net,49701,1369432806997: org.apache.hadoop.hbase.exceptions.YouAreDeadException: Server REPORT rejected; currently processing kiyo.gq1.ygridcore.net,49701,1369432806997 as dead server at org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:338) at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:253) at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:1227) at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:3800) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2103) at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1810) Cause: org.apache.hadoop.hbase.exceptions.YouAreDeadException: org.apache.hadoop.hbase.exceptions.YouAreDeadException: Server REPORT rejected; currently processing kiyo.gq1.ygridcore.net,49701,1369432806997 as dead server at org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:338) at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:253) at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:1227) at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:3800) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2103) at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1810) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:227) at org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:974) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:821) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:158) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:110) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:142) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:337) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1457) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.util.Methods.call(Methods.java:41) at org.apache.hadoop.hbase.security.User.call(User.java:420) at org.apache.hadoop.hbase.security.User.access$300(User.java:51) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:260) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:140) at java.lang.Thread.run(Thread.java:662) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.exceptions.YouAreDeadException): org.apache.hadoop.hbase.exceptions.YouAreDeadException: Server REPORT rejected; currently processing kiyo.gq1.ygridcore.net,49701,1369432806997 as dead server at org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:338) at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:253) at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:1227) at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:3800) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2103) at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1810) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1335) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1539) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1596) at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerReport(RegionServerStatusProtos.java:4095) at org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:972) ... 17 more 2013-05-24 22:00:34,867 INFO [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.HRegionServer(1639): STOPPED: org.apache.hadoop.hbase.exceptions.YouAreDeadException: Server REPORT rejected; currently processing kiyo.gq1.ygridcore.net,49701,1369432806997 as dead server at org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:338) at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:253) at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:1227) at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:3800) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2103) at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1810) 2013-05-24 22:00:34,872 INFO [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.SplitLogWorker(585): Sending interrupt to stop the worker thread 2013-05-24 22:00:34,872 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.SplitLogWorker(267): SplitLogWorker interrupted while waiting for task, exiting: java.lang.InterruptedException 2013-05-24 22:00:34,872 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.SplitLogWorker(199): SplitLogWorker kiyo.gq1.ygridcore.net,49701,1369432806997 exiting 2013-05-24 22:00:34,873 INFO [Thread-155] regionserver.MemStoreFlusher$FlushHandler(267): Thread-155 exiting 2013-05-24 22:00:34,873 INFO [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.logRoller] regionserver.LogRoller(119): LogRoller exiting. 2013-05-24 22:00:34,873 DEBUG [pool-1-thread-1.LruBlockCache.EvictionThread] hfile.LruBlockCache(467): Block cache LRU eviction started; Attempting to free -636.25 MB of total=5.54 MB 2013-05-24 22:00:34,873 INFO [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.compactionChecker] hbase.Chore(93): RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.compactionChecker exiting 2013-05-24 22:00:34,873 INFO [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997] snapshot.RegionServerSnapshotManager(151): Stopping RegionServerSnapshotManager abruptly. 2013-05-24 22:00:34,875 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-0] handler.CloseRegionHandler(125): Processing close of test,yyy,1369432820321.acf78311e603cb0d02309ae08763d196. 2013-05-24 22:00:34,875 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-1] handler.CloseRegionHandler(125): Processing close of test,kkk,1369432820320.1ce9aafaaa422cef4ce0e1895111c733. 2013-05-24 22:00:34,875 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-0] regionserver.HRegion(927): Closing test,yyy,1369432820321.acf78311e603cb0d02309ae08763d196.: disabling compactions & flushes 2013-05-24 22:00:34,875 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-2] handler.CloseRegionHandler(125): Processing close of test,ooo,1369432820320.0d6df81a73ad3b2a79add80716da62ac. 2013-05-24 22:00:34,876 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-1] regionserver.HRegion(927): Closing test,kkk,1369432820320.1ce9aafaaa422cef4ce0e1895111c733.: disabling compactions & flushes 2013-05-24 22:00:34,877 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-2] regionserver.HRegion(927): Closing test,ooo,1369432820320.0d6df81a73ad3b2a79add80716da62ac.: disabling compactions & flushes 2013-05-24 22:00:34,877 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-1] regionserver.HRegion(949): Updates disabled for region test,kkk,1369432820320.1ce9aafaaa422cef4ce0e1895111c733. 2013-05-24 22:00:34,876 INFO [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.HRegionServer(882): aborting server kiyo.gq1.ygridcore.net,49701,1369432806997 2013-05-24 22:00:34,876 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-0] regionserver.HRegion(949): Updates disabled for region test,yyy,1369432820321.acf78311e603cb0d02309ae08763d196. 2013-05-24 22:00:34,877 INFO [StoreCloserThread-test,kkk,1369432820320.1ce9aafaaa422cef4ce0e1895111c733.-1] regionserver.HStore(663): Closed f 2013-05-24 22:00:34,877 INFO [StoreCloserThread-test,yyy,1369432820321.acf78311e603cb0d02309ae08763d196.-1] regionserver.HStore(663): Closed f 2013-05-24 22:00:34,878 INFO [StoreCloserThread-test,kkk,1369432820320.1ce9aafaaa422cef4ce0e1895111c733.-1] regionserver.HStore(663): Closed norep 2013-05-24 22:00:34,878 INFO [StoreCloserThread-test,yyy,1369432820321.acf78311e603cb0d02309ae08763d196.-1] regionserver.HStore(663): Closed norep 2013-05-24 22:00:34,877 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997] catalog.CatalogTracker(208): Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@6774a144 2013-05-24 22:00:34,877 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-2] regionserver.HRegion(949): Updates disabled for region test,ooo,1369432820320.0d6df81a73ad3b2a79add80716da62ac. 2013-05-24 22:00:34,878 INFO [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997] snapshot.RegionServerSnapshotManager(151): Stopping RegionServerSnapshotManager abruptly. 2013-05-24 22:00:34,878 INFO [StoreCloserThread-test,ooo,1369432820320.0d6df81a73ad3b2a79add80716da62ac.-1] regionserver.HStore(663): Closed f 2013-05-24 22:00:34,879 INFO [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.HRegionServer(1048): Waiting on 13 regions to close 2013-05-24 22:00:34,879 INFO [StoreCloserThread-test,ooo,1369432820320.0d6df81a73ad3b2a79add80716da62ac.-1] regionserver.HStore(663): Closed norep 2013-05-24 22:00:34,879 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-1] regionserver.HRegion(1006): Closed test,kkk,1369432820320.1ce9aafaaa422cef4ce0e1895111c733. 2013-05-24 22:00:34,879 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-0] regionserver.HRegion(1006): Closed test,yyy,1369432820321.acf78311e603cb0d02309ae08763d196. 2013-05-24 22:00:34,879 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-2] regionserver.HRegion(1006): Closed test,ooo,1369432820320.0d6df81a73ad3b2a79add80716da62ac. 2013-05-24 22:00:34,880 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-2] handler.CloseRegionHandler(177): Closed region test,ooo,1369432820320.0d6df81a73ad3b2a79add80716da62ac. 2013-05-24 22:00:34,880 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-0] handler.CloseRegionHandler(177): Closed region test,yyy,1369432820321.acf78311e603cb0d02309ae08763d196. 2013-05-24 22:00:34,880 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-2] handler.CloseRegionHandler(125): Processing close of test,zzz,1369432820321.0b4d70d096cb6cbce3376acd563c77dd. 2013-05-24 22:00:34,880 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-0] handler.CloseRegionHandler(125): Processing close of test,fff,1369432820319.f852f52d87b0b7acd213f5b49074dde6. 2013-05-24 22:00:34,881 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-2] regionserver.HRegion(927): Closing test,zzz,1369432820321.0b4d70d096cb6cbce3376acd563c77dd.: disabling compactions & flushes 2013-05-24 22:00:34,881 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-0] regionserver.HRegion(927): Closing test,fff,1369432820319.f852f52d87b0b7acd213f5b49074dde6.: disabling compactions & flushes 2013-05-24 22:00:34,880 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-1] handler.CloseRegionHandler(177): Closed region test,kkk,1369432820320.1ce9aafaaa422cef4ce0e1895111c733. 2013-05-24 22:00:34,881 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-0] regionserver.HRegion(949): Updates disabled for region test,fff,1369432820319.f852f52d87b0b7acd213f5b49074dde6. 2013-05-24 22:00:34,881 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-1] handler.CloseRegionHandler(125): Processing close of test,uuu,1369432820320.dba8b67ade91438cad8a5209736330e6. 2013-05-24 22:00:34,881 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-2] regionserver.HRegion(949): Updates disabled for region test,zzz,1369432820321.0b4d70d096cb6cbce3376acd563c77dd. 2013-05-24 22:00:34,882 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-1] regionserver.HRegion(927): Closing test,uuu,1369432820320.dba8b67ade91438cad8a5209736330e6.: disabling compactions & flushes 2013-05-24 22:00:34,882 INFO [StoreCloserThread-test,fff,1369432820319.f852f52d87b0b7acd213f5b49074dde6.-1] regionserver.HStore(663): Closed f 2013-05-24 22:00:34,882 INFO [StoreCloserThread-test,zzz,1369432820321.0b4d70d096cb6cbce3376acd563c77dd.-1] regionserver.HStore(663): Closed f 2013-05-24 22:00:34,883 INFO [StoreCloserThread-test,zzz,1369432820321.0b4d70d096cb6cbce3376acd563c77dd.-1] regionserver.HStore(663): Closed norep 2013-05-24 22:00:34,882 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-1] regionserver.HRegion(949): Updates disabled for region test,uuu,1369432820320.dba8b67ade91438cad8a5209736330e6. 2013-05-24 22:00:34,883 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-2] regionserver.HRegion(1006): Closed test,zzz,1369432820321.0b4d70d096cb6cbce3376acd563c77dd. 2013-05-24 22:00:34,883 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-2] handler.CloseRegionHandler(177): Closed region test,zzz,1369432820321.0b4d70d096cb6cbce3376acd563c77dd. 2013-05-24 22:00:34,882 INFO [StoreCloserThread-test,fff,1369432820319.f852f52d87b0b7acd213f5b49074dde6.-1] regionserver.HStore(663): Closed norep 2013-05-24 22:00:34,883 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-2] handler.CloseRegionHandler(125): Processing close of test,ggg,1369432820319.4b3ff3efde12c5afe1220e81d155ac73. 2013-05-24 22:00:34,883 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-0] regionserver.HRegion(1006): Closed test,fff,1369432820319.f852f52d87b0b7acd213f5b49074dde6. 2013-05-24 22:00:34,884 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-0] handler.CloseRegionHandler(177): Closed region test,fff,1369432820319.f852f52d87b0b7acd213f5b49074dde6. 2013-05-24 22:00:34,883 INFO [StoreCloserThread-test,uuu,1369432820320.dba8b67ade91438cad8a5209736330e6.-1] regionserver.HStore(663): Closed f 2013-05-24 22:00:34,884 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-0] handler.CloseRegionHandler(125): Processing close of test,sss,1369432820320.872960a61a4ceeb9f64577d8b7cb0ddf. 2013-05-24 22:00:34,884 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-2] regionserver.HRegion(927): Closing test,ggg,1369432820319.4b3ff3efde12c5afe1220e81d155ac73.: disabling compactions & flushes 2013-05-24 22:00:34,884 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-0] regionserver.HRegion(927): Closing test,sss,1369432820320.872960a61a4ceeb9f64577d8b7cb0ddf.: disabling compactions & flushes 2013-05-24 22:00:34,884 INFO [StoreCloserThread-test,uuu,1369432820320.dba8b67ade91438cad8a5209736330e6.-1] regionserver.HStore(663): Closed norep 2013-05-24 22:00:34,885 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-0] regionserver.HRegion(949): Updates disabled for region test,sss,1369432820320.872960a61a4ceeb9f64577d8b7cb0ddf. 2013-05-24 22:00:34,885 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-1] regionserver.HRegion(1006): Closed test,uuu,1369432820320.dba8b67ade91438cad8a5209736330e6. 2013-05-24 22:00:34,885 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-2] regionserver.HRegion(949): Updates disabled for region test,ggg,1369432820319.4b3ff3efde12c5afe1220e81d155ac73. 2013-05-24 22:00:34,885 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-1] handler.CloseRegionHandler(177): Closed region test,uuu,1369432820320.dba8b67ade91438cad8a5209736330e6. 2013-05-24 22:00:34,885 INFO [StoreCloserThread-test,sss,1369432820320.872960a61a4ceeb9f64577d8b7cb0ddf.-1] regionserver.HStore(663): Closed f 2013-05-24 22:00:34,886 INFO [StoreCloserThread-test,ggg,1369432820319.4b3ff3efde12c5afe1220e81d155ac73.-1] regionserver.HStore(663): Closed f 2013-05-24 22:00:34,885 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-1] handler.CloseRegionHandler(125): Processing close of test,,1369432820315.a88e8c9e73f832adb3635d6996d9e72d. 2013-05-24 22:00:34,886 INFO [StoreCloserThread-test,ggg,1369432820319.4b3ff3efde12c5afe1220e81d155ac73.-1] regionserver.HStore(663): Closed norep 2013-05-24 22:00:34,886 INFO [StoreCloserThread-test,sss,1369432820320.872960a61a4ceeb9f64577d8b7cb0ddf.-1] regionserver.HStore(663): Closed norep 2013-05-24 22:00:34,886 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-2] regionserver.HRegion(1006): Closed test,ggg,1369432820319.4b3ff3efde12c5afe1220e81d155ac73. 2013-05-24 22:00:34,887 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-0] regionserver.HRegion(1006): Closed test,sss,1369432820320.872960a61a4ceeb9f64577d8b7cb0ddf. 2013-05-24 22:00:34,887 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-0] handler.CloseRegionHandler(177): Closed region test,sss,1369432820320.872960a61a4ceeb9f64577d8b7cb0ddf. 2013-05-24 22:00:34,887 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-0] handler.CloseRegionHandler(125): Processing close of test,mmm,1369432820320.ab7b9b9773243cff0ac09f3c379e33dc. 2013-05-24 22:00:34,886 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-1] regionserver.HRegion(927): Closing test,,1369432820315.a88e8c9e73f832adb3635d6996d9e72d.: disabling compactions & flushes 2013-05-24 22:00:34,887 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-0] regionserver.HRegion(927): Closing test,mmm,1369432820320.ab7b9b9773243cff0ac09f3c379e33dc.: disabling compactions & flushes 2013-05-24 22:00:34,887 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-2] handler.CloseRegionHandler(177): Closed region test,ggg,1369432820319.4b3ff3efde12c5afe1220e81d155ac73. 2013-05-24 22:00:34,888 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-2] handler.CloseRegionHandler(125): Processing close of test,qqq,1369432820320.ec0b9e20bb8d4dc8fee9b3e80c5cae32. 2013-05-24 22:00:34,887 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-0] regionserver.HRegion(949): Updates disabled for region test,mmm,1369432820320.ab7b9b9773243cff0ac09f3c379e33dc. 2013-05-24 22:00:34,888 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-2] regionserver.HRegion(927): Closing test,qqq,1369432820320.ec0b9e20bb8d4dc8fee9b3e80c5cae32.: disabling compactions & flushes 2013-05-24 22:00:34,887 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-1] regionserver.HRegion(949): Updates disabled for region test,,1369432820315.a88e8c9e73f832adb3635d6996d9e72d. 2013-05-24 22:00:34,888 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-2] regionserver.HRegion(949): Updates disabled for region test,qqq,1369432820320.ec0b9e20bb8d4dc8fee9b3e80c5cae32. 2013-05-24 22:00:34,888 INFO [StoreCloserThread-test,mmm,1369432820320.ab7b9b9773243cff0ac09f3c379e33dc.-1] regionserver.HStore(663): Closed f 2013-05-24 22:00:34,889 INFO [StoreCloserThread-test,qqq,1369432820320.ec0b9e20bb8d4dc8fee9b3e80c5cae32.-1] regionserver.HStore(663): Closed f 2013-05-24 22:00:34,889 INFO [StoreCloserThread-test,mmm,1369432820320.ab7b9b9773243cff0ac09f3c379e33dc.-1] regionserver.HStore(663): Closed norep 2013-05-24 22:00:34,889 INFO [StoreCloserThread-test,qqq,1369432820320.ec0b9e20bb8d4dc8fee9b3e80c5cae32.-1] regionserver.HStore(663): Closed norep 2013-05-24 22:00:34,889 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-0] regionserver.HRegion(1006): Closed test,mmm,1369432820320.ab7b9b9773243cff0ac09f3c379e33dc. 2013-05-24 22:00:34,889 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-2] regionserver.HRegion(1006): Closed test,qqq,1369432820320.ec0b9e20bb8d4dc8fee9b3e80c5cae32. 2013-05-24 22:00:34,888 INFO [StoreCloserThread-test,,1369432820315.a88e8c9e73f832adb3635d6996d9e72d.-1] regionserver.HStore(663): Closed f 2013-05-24 22:00:34,889 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-2] handler.CloseRegionHandler(177): Closed region test,qqq,1369432820320.ec0b9e20bb8d4dc8fee9b3e80c5cae32. 2013-05-24 22:00:34,889 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-0] handler.CloseRegionHandler(177): Closed region test,mmm,1369432820320.ab7b9b9773243cff0ac09f3c379e33dc. 2013-05-24 22:00:34,889 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-2] handler.CloseRegionHandler(125): Processing close of test,nnn,1369432820320.53e3aef4cc8815dd31923659bff5ca21. 2013-05-24 22:00:34,889 INFO [StoreCloserThread-test,,1369432820315.a88e8c9e73f832adb3635d6996d9e72d.-1] regionserver.HStore(663): Closed norep 2013-05-24 22:00:34,890 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-2] regionserver.HRegion(927): Closing test,nnn,1369432820320.53e3aef4cc8815dd31923659bff5ca21.: disabling compactions & flushes 2013-05-24 22:00:34,890 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-0] handler.CloseRegionHandler(125): Processing close of test,jjj,1369432820320.eb38c552d2a930aa1ce862202da80908. 2013-05-24 22:00:34,890 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-2] regionserver.HRegion(949): Updates disabled for region test,nnn,1369432820320.53e3aef4cc8815dd31923659bff5ca21. 2013-05-24 22:00:34,890 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-1] regionserver.HRegion(1006): Closed test,,1369432820315.a88e8c9e73f832adb3635d6996d9e72d. 2013-05-24 22:00:34,891 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-0] regionserver.HRegion(927): Closing test,jjj,1369432820320.eb38c552d2a930aa1ce862202da80908.: disabling compactions & flushes 2013-05-24 22:00:34,891 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-1] handler.CloseRegionHandler(177): Closed region test,,1369432820315.a88e8c9e73f832adb3635d6996d9e72d. 2013-05-24 22:00:34,891 INFO [StoreCloserThread-test,nnn,1369432820320.53e3aef4cc8815dd31923659bff5ca21.-1] regionserver.HStore(663): Closed f 2013-05-24 22:00:34,891 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-0] regionserver.HRegion(949): Updates disabled for region test,jjj,1369432820320.eb38c552d2a930aa1ce862202da80908. 2013-05-24 22:00:34,892 INFO [StoreCloserThread-test,nnn,1369432820320.53e3aef4cc8815dd31923659bff5ca21.-1] regionserver.HStore(663): Closed norep 2013-05-24 22:00:34,892 INFO [StoreCloserThread-test,jjj,1369432820320.eb38c552d2a930aa1ce862202da80908.-1] regionserver.HStore(663): Closed f 2013-05-24 22:00:34,892 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-2] regionserver.HRegion(1006): Closed test,nnn,1369432820320.53e3aef4cc8815dd31923659bff5ca21. 2013-05-24 22:00:34,892 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-2] handler.CloseRegionHandler(177): Closed region test,nnn,1369432820320.53e3aef4cc8815dd31923659bff5ca21. 2013-05-24 22:00:34,892 INFO [StoreCloserThread-test,jjj,1369432820320.eb38c552d2a930aa1ce862202da80908.-1] regionserver.HStore(663): Closed norep 2013-05-24 22:00:34,892 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-0] regionserver.HRegion(1006): Closed test,jjj,1369432820320.eb38c552d2a930aa1ce862202da80908. 2013-05-24 22:00:34,893 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,49701,1369432806997-0] handler.CloseRegionHandler(177): Closed region test,jjj,1369432820320.eb38c552d2a930aa1ce862202da80908. 2013-05-24 22:00:34,893 INFO [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.periodicFlusher] hbase.Chore(93): RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.periodicFlusher exiting 2013-05-24 22:00:34,895 INFO [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.leaseChecker] regionserver.Leases(124): RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.leaseChecker closing leases 2013-05-24 22:00:34,895 INFO [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.leaseChecker] regionserver.Leases(131): RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.leaseChecker closed leases 2013-05-24 22:00:34,905 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432834215 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:34,910 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:34,924 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.SplitLogManager(1117): Mark region 53e3aef4cc8815dd31923659bff5ca21 recovering from failed region server kiyo.gq1.ygridcore.net,49701,1369432806997 2013-05-24 22:00:34,933 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] zookeeper.ZKUtil(690): master:49101-0x13ed88e2cdf0004 Unable to get data of znode /1/recovering-regions/0d6df81a73ad3b2a79add80716da62ac because node does not exist (not an error) 2013-05-24 22:00:34,934 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432834215 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:34,938 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:34,959 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432834215 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:34,964 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:35,028 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.SplitLogManager(1117): Mark region 0d6df81a73ad3b2a79add80716da62ac recovering from failed region server kiyo.gq1.ygridcore.net,49701,1369432806997 2013-05-24 22:00:35,029 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] zookeeper.ZKUtil(690): master:49101-0x13ed88e2cdf0004 Unable to get data of znode /1/recovering-regions/ec0b9e20bb8d4dc8fee9b3e80c5cae32 because node does not exist (not an error) 2013-05-24 22:00:35,079 INFO [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.HRegionServer(912): stopping server kiyo.gq1.ygridcore.net,49701,1369432806997; all regions closed. 2013-05-24 22:00:35,079 INFO [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.logSyncer] wal.FSHLog$LogSyncer(967): RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.logSyncer exiting 2013-05-24 22:00:35,080 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997] wal.FSHLog(809): closing hlog writer in hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997 2013-05-24 22:00:35,083 INFO [IPC Server handler 8 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53333 is added to blk_2036498173552211874_1041{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:59993|RBW], ReplicaUnderConstruction[127.0.0.1:53333|RBW]]} size 0 2013-05-24 22:00:35,084 INFO [IPC Server handler 3 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59993 is added to blk_2036498173552211874_1041 size 23036 2013-05-24 22:00:35,130 WARN [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] zookeeper.RecoverableZooKeeper(237): Possibly transient ZooKeeper, quorum=localhost:51411, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /1/replication/rs/kiyo.gq1.ygridcore.net,49701,1369432806997/2/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432832342 2013-05-24 22:00:35,131 INFO [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] util.RetryCounter(54): Sleeping 20ms before retry #1... 2013-05-24 22:00:35,282 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432834215 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:35,286 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:35,304 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.SplitLogManager(1117): Mark region ec0b9e20bb8d4dc8fee9b3e80c5cae32 recovering from failed region server kiyo.gq1.ygridcore.net,49701,1369432806997 2013-05-24 22:00:35,313 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] zookeeper.ZKUtil(690): master:49101-0x13ed88e2cdf0004 Unable to get data of znode /1/recovering-regions/872960a61a4ceeb9f64577d8b7cb0ddf because node does not exist (not an error) 2013-05-24 22:00:35,316 DEBUG [Thread-584] client.MetaScanner(160): Scanning .META. starting at row=test,sss,00000000000000 for max=10 with caching=10 2013-05-24 22:00:35,316 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432834215 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:35,316 DEBUG [Thread-584] client.ClientScanner(96): Creating scanner over .META. starting at key 'test,sss,00000000000000' 2013-05-24 22:00:35,317 DEBUG [Thread-584] client.ClientScanner(209): Advancing internal scanner to startKey at 'test,sss,00000000000000' 2013-05-24 22:00:35,321 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:35,325 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(583): Nothing to replicate, sleeping 100 times 1 2013-05-24 22:00:35,329 DEBUG [Thread-584] client.ClientScanner(196): Finished scanning region {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,} 2013-05-24 22:00:35,329 DEBUG [Thread-584] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,sss,1369432820320.872960a61a4ceeb9f64577d8b7cb0ddf. is kiyo.gq1.ygridcore.net:49701 2013-05-24 22:00:35,330 DEBUG [Thread-584] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,uuu,1369432820320.dba8b67ade91438cad8a5209736330e6. is kiyo.gq1.ygridcore.net:49701 2013-05-24 22:00:35,331 DEBUG [Thread-584] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,vvv,1369432820320.84d2bf1edb7e9df4777e80207fc739d2. is kiyo.gq1.ygridcore.net:42690 2013-05-24 22:00:35,331 DEBUG [Thread-584] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,www,1369432820320.643c76d94d70be44d35b5a12853bb852. is kiyo.gq1.ygridcore.net:42690 2013-05-24 22:00:35,332 DEBUG [Thread-584] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,xxx,1369432820320.b03c7b8b6013fa49f346edeb3a7cad87. is kiyo.gq1.ygridcore.net:42690 2013-05-24 22:00:35,333 DEBUG [Thread-584] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,yyy,1369432820321.acf78311e603cb0d02309ae08763d196. is kiyo.gq1.ygridcore.net:49701 2013-05-24 22:00:35,333 DEBUG [Thread-584] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,zzz,1369432820321.0b4d70d096cb6cbce3376acd563c77dd. is kiyo.gq1.ygridcore.net:49701 2013-05-24 22:00:35,346 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.SplitLogManager(1117): Mark region 872960a61a4ceeb9f64577d8b7cb0ddf recovering from failed region server kiyo.gq1.ygridcore.net,49701,1369432806997 2013-05-24 22:00:35,347 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] zookeeper.ZKUtil(690): master:49101-0x13ed88e2cdf0004 Unable to get data of znode /1/recovering-regions/dba8b67ade91438cad8a5209736330e6 because node does not exist (not an error) 2013-05-24 22:00:35,393 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.SplitLogManager(1117): Mark region dba8b67ade91438cad8a5209736330e6 recovering from failed region server kiyo.gq1.ygridcore.net,49701,1369432806997 2013-05-24 22:00:35,394 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] zookeeper.ZKUtil(690): master:49101-0x13ed88e2cdf0004 Unable to get data of znode /1/recovering-regions/acf78311e603cb0d02309ae08763d196 because node does not exist (not an error) 2013-05-24 22:00:35,399 INFO [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.Leases(124): RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997 closing leases 2013-05-24 22:00:35,400 INFO [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.Leases(131): RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997 closed leases 2013-05-24 22:00:35,400 INFO [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.CompactSplitThread(318): Waiting for Split Thread to finish... 2013-05-24 22:00:35,400 INFO [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.CompactSplitThread(318): Waiting for Merge Thread to finish... 2013-05-24 22:00:35,400 INFO [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.CompactSplitThread(318): Waiting for Large Compaction Thread to finish... 2013-05-24 22:00:35,400 INFO [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.CompactSplitThread(318): Waiting for Small Compaction Thread to finish... 2013-05-24 22:00:35,401 INFO [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.ReplicationSource(761): Closing source 2 because: Region server is closing 2013-05-24 22:00:35,429 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432834215 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:35,433 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:35,435 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(583): Nothing to replicate, sleeping 100 times 2 2013-05-24 22:00:35,576 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.SplitLogManager(1117): Mark region acf78311e603cb0d02309ae08763d196 recovering from failed region server kiyo.gq1.ygridcore.net,49701,1369432806997 2013-05-24 22:00:35,577 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] zookeeper.ZKUtil(690): master:49101-0x13ed88e2cdf0004 Unable to get data of znode /1/recovering-regions/0b4d70d096cb6cbce3376acd563c77dd because node does not exist (not an error) 2013-05-24 22:00:35,637 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432834215 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:35,642 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:35,646 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(583): Nothing to replicate, sleeping 100 times 3 2013-05-24 22:00:35,895 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.SplitLogManager(1117): Mark region 0b4d70d096cb6cbce3376acd563c77dd recovering from failed region server kiyo.gq1.ygridcore.net,49701,1369432806997 2013-05-24 22:00:35,895 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] handler.ServerShutdownHandler(205): Reassigning 13 region(s) that kiyo.gq1.ygridcore.net,49701,1369432806997 was carrying (and 0 regions(s) that were opening on this server) 2013-05-24 22:00:35,896 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.AssignmentManager(2411): Not use bulk assigning since we are assigning only 13 region(s) to 1 server(s) 2013-05-24 22:00:35,896 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.AssignmentManager(1474): Bulk assigning 13 region(s) to kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:35,896 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.RegionStates(325): Offline a region with current state=OPEN, expected state=OFFLINE, assigned to server: kiyo.gq1.ygridcore.net,49701,1369432806997, expected null 2013-05-24 22:00:35,896 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.RegionStates(264): Region {NAME => 'test,,1369432820315.a88e8c9e73f832adb3635d6996d9e72d.', STARTKEY => '', ENDKEY => 'bbb', ENCODED => a88e8c9e73f832adb3635d6996d9e72d,} transitioned from {test,,1369432820315.a88e8c9e73f832adb3635d6996d9e72d. state=OPEN, ts=1369432821909, server=kiyo.gq1.ygridcore.net,49701,1369432806997} to {test,,1369432820315.a88e8c9e73f832adb3635d6996d9e72d. state=OFFLINE, ts=1369432835896, server=null} 2013-05-24 22:00:35,896 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.RegionStates(264): Region {NAME => 'test,,1369432820315.a88e8c9e73f832adb3635d6996d9e72d.', STARTKEY => '', ENDKEY => 'bbb', ENCODED => a88e8c9e73f832adb3635d6996d9e72d,} transitioned from {test,,1369432820315.a88e8c9e73f832adb3635d6996d9e72d. state=OFFLINE, ts=1369432835896, server=null} to {test,,1369432820315.a88e8c9e73f832adb3635d6996d9e72d. state=OFFLINE, ts=1369432835896, server=null} 2013-05-24 22:00:35,897 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] zookeeper.ZKAssign(177): master:49101-0x13ed88e2cdf0004 Async create of unassigned node for a88e8c9e73f832adb3635d6996d9e72d with OFFLINE state 2013-05-24 22:00:35,897 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.RegionStates(325): Offline a region with current state=OPEN, expected state=OFFLINE, assigned to server: kiyo.gq1.ygridcore.net,49701,1369432806997, expected null 2013-05-24 22:00:35,897 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.RegionStates(264): Region {NAME => 'test,fff,1369432820319.f852f52d87b0b7acd213f5b49074dde6.', STARTKEY => 'fff', ENDKEY => 'ggg', ENCODED => f852f52d87b0b7acd213f5b49074dde6,} transitioned from {test,fff,1369432820319.f852f52d87b0b7acd213f5b49074dde6. state=OPEN, ts=1369432822063, server=kiyo.gq1.ygridcore.net,49701,1369432806997} to {test,fff,1369432820319.f852f52d87b0b7acd213f5b49074dde6. state=OFFLINE, ts=1369432835897, server=null} 2013-05-24 22:00:35,897 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.RegionStates(264): Region {NAME => 'test,fff,1369432820319.f852f52d87b0b7acd213f5b49074dde6.', STARTKEY => 'fff', ENDKEY => 'ggg', ENCODED => f852f52d87b0b7acd213f5b49074dde6,} transitioned from {test,fff,1369432820319.f852f52d87b0b7acd213f5b49074dde6. state=OFFLINE, ts=1369432835897, server=null} to {test,fff,1369432820319.f852f52d87b0b7acd213f5b49074dde6. state=OFFLINE, ts=1369432835897, server=null} 2013-05-24 22:00:35,897 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] zookeeper.ZKAssign(177): master:49101-0x13ed88e2cdf0004 Async create of unassigned node for f852f52d87b0b7acd213f5b49074dde6 with OFFLINE state 2013-05-24 22:00:35,898 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.RegionStates(325): Offline a region with current state=OPEN, expected state=OFFLINE, assigned to server: kiyo.gq1.ygridcore.net,49701,1369432806997, expected null 2013-05-24 22:00:35,898 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.RegionStates(264): Region {NAME => 'test,ggg,1369432820319.4b3ff3efde12c5afe1220e81d155ac73.', STARTKEY => 'ggg', ENDKEY => 'hhh', ENCODED => 4b3ff3efde12c5afe1220e81d155ac73,} transitioned from {test,ggg,1369432820319.4b3ff3efde12c5afe1220e81d155ac73. state=OPEN, ts=1369432822199, server=kiyo.gq1.ygridcore.net,49701,1369432806997} to {test,ggg,1369432820319.4b3ff3efde12c5afe1220e81d155ac73. state=OFFLINE, ts=1369432835898, server=null} 2013-05-24 22:00:35,898 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.RegionStates(264): Region {NAME => 'test,ggg,1369432820319.4b3ff3efde12c5afe1220e81d155ac73.', STARTKEY => 'ggg', ENDKEY => 'hhh', ENCODED => 4b3ff3efde12c5afe1220e81d155ac73,} transitioned from {test,ggg,1369432820319.4b3ff3efde12c5afe1220e81d155ac73. state=OFFLINE, ts=1369432835898, server=null} to {test,ggg,1369432820319.4b3ff3efde12c5afe1220e81d155ac73. state=OFFLINE, ts=1369432835898, server=null} 2013-05-24 22:00:35,898 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] zookeeper.ZKAssign(177): master:49101-0x13ed88e2cdf0004 Async create of unassigned node for 4b3ff3efde12c5afe1220e81d155ac73 with OFFLINE state 2013-05-24 22:00:35,898 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.RegionStates(325): Offline a region with current state=OPEN, expected state=OFFLINE, assigned to server: kiyo.gq1.ygridcore.net,49701,1369432806997, expected null 2013-05-24 22:00:35,898 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.RegionStates(264): Region {NAME => 'test,jjj,1369432820320.eb38c552d2a930aa1ce862202da80908.', STARTKEY => 'jjj', ENDKEY => 'kkk', ENCODED => eb38c552d2a930aa1ce862202da80908,} transitioned from {test,jjj,1369432820320.eb38c552d2a930aa1ce862202da80908. state=OPEN, ts=1369432822090, server=kiyo.gq1.ygridcore.net,49701,1369432806997} to {test,jjj,1369432820320.eb38c552d2a930aa1ce862202da80908. state=OFFLINE, ts=1369432835898, server=null} 2013-05-24 22:00:35,899 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.RegionStates(264): Region {NAME => 'test,jjj,1369432820320.eb38c552d2a930aa1ce862202da80908.', STARTKEY => 'jjj', ENDKEY => 'kkk', ENCODED => eb38c552d2a930aa1ce862202da80908,} transitioned from {test,jjj,1369432820320.eb38c552d2a930aa1ce862202da80908. state=OFFLINE, ts=1369432835898, server=null} to {test,jjj,1369432820320.eb38c552d2a930aa1ce862202da80908. state=OFFLINE, ts=1369432835899, server=null} 2013-05-24 22:00:35,899 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] zookeeper.ZKAssign(177): master:49101-0x13ed88e2cdf0004 Async create of unassigned node for eb38c552d2a930aa1ce862202da80908 with OFFLINE state 2013-05-24 22:00:35,899 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.RegionStates(325): Offline a region with current state=OPEN, expected state=OFFLINE, assigned to server: kiyo.gq1.ygridcore.net,49701,1369432806997, expected null 2013-05-24 22:00:35,899 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.RegionStates(264): Region {NAME => 'test,kkk,1369432820320.1ce9aafaaa422cef4ce0e1895111c733.', STARTKEY => 'kkk', ENDKEY => 'lll', ENCODED => 1ce9aafaaa422cef4ce0e1895111c733,} transitioned from {test,kkk,1369432820320.1ce9aafaaa422cef4ce0e1895111c733. state=OPEN, ts=1369432822697, server=kiyo.gq1.ygridcore.net,49701,1369432806997} to {test,kkk,1369432820320.1ce9aafaaa422cef4ce0e1895111c733. state=OFFLINE, ts=1369432835899, server=null} 2013-05-24 22:00:35,899 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.RegionStates(264): Region {NAME => 'test,kkk,1369432820320.1ce9aafaaa422cef4ce0e1895111c733.', STARTKEY => 'kkk', ENDKEY => 'lll', ENCODED => 1ce9aafaaa422cef4ce0e1895111c733,} transitioned from {test,kkk,1369432820320.1ce9aafaaa422cef4ce0e1895111c733. state=OFFLINE, ts=1369432835899, server=null} to {test,kkk,1369432820320.1ce9aafaaa422cef4ce0e1895111c733. state=OFFLINE, ts=1369432835899, server=null} 2013-05-24 22:00:35,899 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] zookeeper.ZKAssign(177): master:49101-0x13ed88e2cdf0004 Async create of unassigned node for 1ce9aafaaa422cef4ce0e1895111c733 with OFFLINE state 2013-05-24 22:00:35,900 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.RegionStates(325): Offline a region with current state=OPEN, expected state=OFFLINE, assigned to server: kiyo.gq1.ygridcore.net,49701,1369432806997, expected null 2013-05-24 22:00:35,900 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.RegionStates(264): Region {NAME => 'test,mmm,1369432820320.ab7b9b9773243cff0ac09f3c379e33dc.', STARTKEY => 'mmm', ENDKEY => 'nnn', ENCODED => ab7b9b9773243cff0ac09f3c379e33dc,} transitioned from {test,mmm,1369432820320.ab7b9b9773243cff0ac09f3c379e33dc. state=OPEN, ts=1369432822733, server=kiyo.gq1.ygridcore.net,49701,1369432806997} to {test,mmm,1369432820320.ab7b9b9773243cff0ac09f3c379e33dc. state=OFFLINE, ts=1369432835900, server=null} 2013-05-24 22:00:35,900 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.RegionStates(264): Region {NAME => 'test,mmm,1369432820320.ab7b9b9773243cff0ac09f3c379e33dc.', STARTKEY => 'mmm', ENDKEY => 'nnn', ENCODED => ab7b9b9773243cff0ac09f3c379e33dc,} transitioned from {test,mmm,1369432820320.ab7b9b9773243cff0ac09f3c379e33dc. state=OFFLINE, ts=1369432835900, server=null} to {test,mmm,1369432820320.ab7b9b9773243cff0ac09f3c379e33dc. state=OFFLINE, ts=1369432835900, server=null} 2013-05-24 22:00:35,900 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] zookeeper.ZKAssign(177): master:49101-0x13ed88e2cdf0004 Async create of unassigned node for ab7b9b9773243cff0ac09f3c379e33dc with OFFLINE state 2013-05-24 22:00:35,900 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.RegionStates(325): Offline a region with current state=OPEN, expected state=OFFLINE, assigned to server: kiyo.gq1.ygridcore.net,49701,1369432806997, expected null 2013-05-24 22:00:35,900 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.RegionStates(264): Region {NAME => 'test,nnn,1369432820320.53e3aef4cc8815dd31923659bff5ca21.', STARTKEY => 'nnn', ENDKEY => 'ooo', ENCODED => 53e3aef4cc8815dd31923659bff5ca21,} transitioned from {test,nnn,1369432820320.53e3aef4cc8815dd31923659bff5ca21. state=OPEN, ts=1369432824096, server=kiyo.gq1.ygridcore.net,49701,1369432806997} to {test,nnn,1369432820320.53e3aef4cc8815dd31923659bff5ca21. state=OFFLINE, ts=1369432835900, server=null} 2013-05-24 22:00:35,901 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.RegionStates(264): Region {NAME => 'test,nnn,1369432820320.53e3aef4cc8815dd31923659bff5ca21.', STARTKEY => 'nnn', ENDKEY => 'ooo', ENCODED => 53e3aef4cc8815dd31923659bff5ca21,} transitioned from {test,nnn,1369432820320.53e3aef4cc8815dd31923659bff5ca21. state=OFFLINE, ts=1369432835900, server=null} to {test,nnn,1369432820320.53e3aef4cc8815dd31923659bff5ca21. state=OFFLINE, ts=1369432835901, server=null} 2013-05-24 22:00:35,901 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] zookeeper.ZKAssign(177): master:49101-0x13ed88e2cdf0004 Async create of unassigned node for 53e3aef4cc8815dd31923659bff5ca21 with OFFLINE state 2013-05-24 22:00:35,901 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.RegionStates(325): Offline a region with current state=OPEN, expected state=OFFLINE, assigned to server: kiyo.gq1.ygridcore.net,49701,1369432806997, expected null 2013-05-24 22:00:35,901 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.RegionStates(264): Region {NAME => 'test,ooo,1369432820320.0d6df81a73ad3b2a79add80716da62ac.', STARTKEY => 'ooo', ENDKEY => 'ppp', ENCODED => 0d6df81a73ad3b2a79add80716da62ac,} transitioned from {test,ooo,1369432820320.0d6df81a73ad3b2a79add80716da62ac. state=OPEN, ts=1369432824665, server=kiyo.gq1.ygridcore.net,49701,1369432806997} to {test,ooo,1369432820320.0d6df81a73ad3b2a79add80716da62ac. state=OFFLINE, ts=1369432835901, server=null} 2013-05-24 22:00:35,901 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.RegionStates(264): Region {NAME => 'test,ooo,1369432820320.0d6df81a73ad3b2a79add80716da62ac.', STARTKEY => 'ooo', ENDKEY => 'ppp', ENCODED => 0d6df81a73ad3b2a79add80716da62ac,} transitioned from {test,ooo,1369432820320.0d6df81a73ad3b2a79add80716da62ac. state=OFFLINE, ts=1369432835901, server=null} to {test,ooo,1369432820320.0d6df81a73ad3b2a79add80716da62ac. state=OFFLINE, ts=1369432835901, server=null} 2013-05-24 22:00:35,902 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] zookeeper.ZKAssign(177): master:49101-0x13ed88e2cdf0004 Async create of unassigned node for 0d6df81a73ad3b2a79add80716da62ac with OFFLINE state 2013-05-24 22:00:35,902 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.RegionStates(325): Offline a region with current state=OPEN, expected state=OFFLINE, assigned to server: kiyo.gq1.ygridcore.net,49701,1369432806997, expected null 2013-05-24 22:00:35,902 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.RegionStates(264): Region {NAME => 'test,qqq,1369432820320.ec0b9e20bb8d4dc8fee9b3e80c5cae32.', STARTKEY => 'qqq', ENDKEY => 'rrr', ENCODED => ec0b9e20bb8d4dc8fee9b3e80c5cae32,} transitioned from {test,qqq,1369432820320.ec0b9e20bb8d4dc8fee9b3e80c5cae32. state=OPEN, ts=1369432824242, server=kiyo.gq1.ygridcore.net,49701,1369432806997} to {test,qqq,1369432820320.ec0b9e20bb8d4dc8fee9b3e80c5cae32. state=OFFLINE, ts=1369432835902, server=null} 2013-05-24 22:00:35,902 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.RegionStates(264): Region {NAME => 'test,qqq,1369432820320.ec0b9e20bb8d4dc8fee9b3e80c5cae32.', STARTKEY => 'qqq', ENDKEY => 'rrr', ENCODED => ec0b9e20bb8d4dc8fee9b3e80c5cae32,} transitioned from {test,qqq,1369432820320.ec0b9e20bb8d4dc8fee9b3e80c5cae32. state=OFFLINE, ts=1369432835902, server=null} to {test,qqq,1369432820320.ec0b9e20bb8d4dc8fee9b3e80c5cae32. state=OFFLINE, ts=1369432835902, server=null} 2013-05-24 22:00:35,902 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] zookeeper.ZKAssign(177): master:49101-0x13ed88e2cdf0004 Async create of unassigned node for ec0b9e20bb8d4dc8fee9b3e80c5cae32 with OFFLINE state 2013-05-24 22:00:35,902 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.RegionStates(325): Offline a region with current state=OPEN, expected state=OFFLINE, assigned to server: kiyo.gq1.ygridcore.net,49701,1369432806997, expected null 2013-05-24 22:00:35,903 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.RegionStates(264): Region {NAME => 'test,sss,1369432820320.872960a61a4ceeb9f64577d8b7cb0ddf.', STARTKEY => 'sss', ENDKEY => 'ttt', ENCODED => 872960a61a4ceeb9f64577d8b7cb0ddf,} transitioned from {test,sss,1369432820320.872960a61a4ceeb9f64577d8b7cb0ddf. state=OPEN, ts=1369432824260, server=kiyo.gq1.ygridcore.net,49701,1369432806997} to {test,sss,1369432820320.872960a61a4ceeb9f64577d8b7cb0ddf. state=OFFLINE, ts=1369432835903, server=null} 2013-05-24 22:00:35,903 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.RegionStates(264): Region {NAME => 'test,sss,1369432820320.872960a61a4ceeb9f64577d8b7cb0ddf.', STARTKEY => 'sss', ENDKEY => 'ttt', ENCODED => 872960a61a4ceeb9f64577d8b7cb0ddf,} transitioned from {test,sss,1369432820320.872960a61a4ceeb9f64577d8b7cb0ddf. state=OFFLINE, ts=1369432835903, server=null} to {test,sss,1369432820320.872960a61a4ceeb9f64577d8b7cb0ddf. state=OFFLINE, ts=1369432835903, server=null} 2013-05-24 22:00:35,903 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] zookeeper.ZKAssign(177): master:49101-0x13ed88e2cdf0004 Async create of unassigned node for 872960a61a4ceeb9f64577d8b7cb0ddf with OFFLINE state 2013-05-24 22:00:35,903 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.RegionStates(325): Offline a region with current state=OPEN, expected state=OFFLINE, assigned to server: kiyo.gq1.ygridcore.net,49701,1369432806997, expected null 2013-05-24 22:00:35,903 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.RegionStates(264): Region {NAME => 'test,uuu,1369432820320.dba8b67ade91438cad8a5209736330e6.', STARTKEY => 'uuu', ENDKEY => 'vvv', ENCODED => dba8b67ade91438cad8a5209736330e6,} transitioned from {test,uuu,1369432820320.dba8b67ade91438cad8a5209736330e6. state=OPEN, ts=1369432824479, server=kiyo.gq1.ygridcore.net,49701,1369432806997} to {test,uuu,1369432820320.dba8b67ade91438cad8a5209736330e6. state=OFFLINE, ts=1369432835903, server=null} 2013-05-24 22:00:35,903 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.RegionStates(264): Region {NAME => 'test,uuu,1369432820320.dba8b67ade91438cad8a5209736330e6.', STARTKEY => 'uuu', ENDKEY => 'vvv', ENCODED => dba8b67ade91438cad8a5209736330e6,} transitioned from {test,uuu,1369432820320.dba8b67ade91438cad8a5209736330e6. state=OFFLINE, ts=1369432835903, server=null} to {test,uuu,1369432820320.dba8b67ade91438cad8a5209736330e6. state=OFFLINE, ts=1369432835903, server=null} 2013-05-24 22:00:35,904 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] zookeeper.ZKAssign(177): master:49101-0x13ed88e2cdf0004 Async create of unassigned node for dba8b67ade91438cad8a5209736330e6 with OFFLINE state 2013-05-24 22:00:35,904 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.RegionStates(325): Offline a region with current state=OPEN, expected state=OFFLINE, assigned to server: kiyo.gq1.ygridcore.net,49701,1369432806997, expected null 2013-05-24 22:00:35,904 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.RegionStates(264): Region {NAME => 'test,yyy,1369432820321.acf78311e603cb0d02309ae08763d196.', STARTKEY => 'yyy', ENDKEY => 'zzz', ENCODED => acf78311e603cb0d02309ae08763d196,} transitioned from {test,yyy,1369432820321.acf78311e603cb0d02309ae08763d196. state=OPEN, ts=1369432824716, server=kiyo.gq1.ygridcore.net,49701,1369432806997} to {test,yyy,1369432820321.acf78311e603cb0d02309ae08763d196. state=OFFLINE, ts=1369432835904, server=null} 2013-05-24 22:00:35,904 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.RegionStates(264): Region {NAME => 'test,yyy,1369432820321.acf78311e603cb0d02309ae08763d196.', STARTKEY => 'yyy', ENDKEY => 'zzz', ENCODED => acf78311e603cb0d02309ae08763d196,} transitioned from {test,yyy,1369432820321.acf78311e603cb0d02309ae08763d196. state=OFFLINE, ts=1369432835904, server=null} to {test,yyy,1369432820321.acf78311e603cb0d02309ae08763d196. state=OFFLINE, ts=1369432835904, server=null} 2013-05-24 22:00:35,904 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] zookeeper.ZKAssign(177): master:49101-0x13ed88e2cdf0004 Async create of unassigned node for acf78311e603cb0d02309ae08763d196 with OFFLINE state 2013-05-24 22:00:35,904 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.RegionStates(325): Offline a region with current state=OPEN, expected state=OFFLINE, assigned to server: kiyo.gq1.ygridcore.net,49701,1369432806997, expected null 2013-05-24 22:00:35,906 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.RegionStates(264): Region {NAME => 'test,zzz,1369432820321.0b4d70d096cb6cbce3376acd563c77dd.', STARTKEY => 'zzz', ENDKEY => '', ENCODED => 0b4d70d096cb6cbce3376acd563c77dd,} transitioned from {test,zzz,1369432820321.0b4d70d096cb6cbce3376acd563c77dd. state=OPEN, ts=1369432824893, server=kiyo.gq1.ygridcore.net,49701,1369432806997} to {test,zzz,1369432820321.0b4d70d096cb6cbce3376acd563c77dd. state=OFFLINE, ts=1369432835906, server=null} 2013-05-24 22:00:35,906 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.RegionStates(264): Region {NAME => 'test,zzz,1369432820321.0b4d70d096cb6cbce3376acd563c77dd.', STARTKEY => 'zzz', ENDKEY => '', ENCODED => 0b4d70d096cb6cbce3376acd563c77dd,} transitioned from {test,zzz,1369432820321.0b4d70d096cb6cbce3376acd563c77dd. state=OFFLINE, ts=1369432835906, server=null} to {test,zzz,1369432820321.0b4d70d096cb6cbce3376acd563c77dd. state=OFFLINE, ts=1369432835906, server=null} 2013-05-24 22:00:35,907 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] zookeeper.ZKAssign(177): master:49101-0x13ed88e2cdf0004 Async create of unassigned node for 0b4d70d096cb6cbce3376acd563c77dd with OFFLINE state 2013-05-24 22:00:35,929 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/region-in-transition 2013-05-24 22:00:35,929 DEBUG [pool-1-thread-1-EventThread] master.OfflineCallback(69): rs={test,,1369432820315.a88e8c9e73f832adb3635d6996d9e72d. state=OFFLINE, ts=1369432835896, server=null}, server=kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:35,946 DEBUG [pool-1-thread-1-EventThread] master.OfflineCallback(69): rs={test,fff,1369432820319.f852f52d87b0b7acd213f5b49074dde6. state=OFFLINE, ts=1369432835897, server=null}, server=kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:35,946 DEBUG [pool-1-thread-1-EventThread] master.OfflineCallback(69): rs={test,ggg,1369432820319.4b3ff3efde12c5afe1220e81d155ac73. state=OFFLINE, ts=1369432835898, server=null}, server=kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:35,946 DEBUG [pool-1-thread-1-EventThread] master.OfflineCallback(69): rs={test,jjj,1369432820320.eb38c552d2a930aa1ce862202da80908. state=OFFLINE, ts=1369432835898, server=null}, server=kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:35,947 DEBUG [pool-1-thread-1-EventThread] master.OfflineCallback(69): rs={test,kkk,1369432820320.1ce9aafaaa422cef4ce0e1895111c733. state=OFFLINE, ts=1369432835899, server=null}, server=kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:35,947 DEBUG [pool-1-thread-1-EventThread] master.OfflineCallback(69): rs={test,mmm,1369432820320.ab7b9b9773243cff0ac09f3c379e33dc. state=OFFLINE, ts=1369432835900, server=null}, server=kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:35,948 DEBUG [pool-1-thread-1-EventThread] master.OfflineCallback(69): rs={test,nnn,1369432820320.53e3aef4cc8815dd31923659bff5ca21. state=OFFLINE, ts=1369432835900, server=null}, server=kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:35,948 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432834215 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:35,948 DEBUG [pool-1-thread-1-EventThread] master.OfflineCallback(69): rs={test,ooo,1369432820320.0d6df81a73ad3b2a79add80716da62ac. state=OFFLINE, ts=1369432835901, server=null}, server=kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:35,948 DEBUG [pool-1-thread-1-EventThread] master.OfflineCallback(69): rs={test,qqq,1369432820320.ec0b9e20bb8d4dc8fee9b3e80c5cae32. state=OFFLINE, ts=1369432835902, server=null}, server=kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:35,949 DEBUG [pool-1-thread-1-EventThread] master.OfflineCallback(69): rs={test,sss,1369432820320.872960a61a4ceeb9f64577d8b7cb0ddf. state=OFFLINE, ts=1369432835903, server=null}, server=kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:35,949 DEBUG [pool-1-thread-1-EventThread] master.OfflineCallback(69): rs={test,uuu,1369432820320.dba8b67ade91438cad8a5209736330e6. state=OFFLINE, ts=1369432835903, server=null}, server=kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:35,950 DEBUG [pool-1-thread-1-EventThread] master.OfflineCallback(69): rs={test,yyy,1369432820321.acf78311e603cb0d02309ae08763d196. state=OFFLINE, ts=1369432835904, server=null}, server=kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:35,950 DEBUG [pool-1-thread-1-EventThread] master.OfflineCallback(69): rs={test,zzz,1369432820321.0b4d70d096cb6cbce3376acd563c77dd. state=OFFLINE, ts=1369432835906, server=null}, server=kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:35,951 DEBUG [pool-1-thread-1-EventThread] master.OfflineCallback$ExistCallback(106): rs={test,,1369432820315.a88e8c9e73f832adb3635d6996d9e72d. state=OFFLINE, ts=1369432835896, server=null}, server=kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:35,951 DEBUG [pool-1-thread-1-EventThread] master.OfflineCallback$ExistCallback(106): rs={test,fff,1369432820319.f852f52d87b0b7acd213f5b49074dde6. state=OFFLINE, ts=1369432835897, server=null}, server=kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:35,951 DEBUG [pool-1-thread-1-EventThread] master.OfflineCallback$ExistCallback(106): rs={test,ggg,1369432820319.4b3ff3efde12c5afe1220e81d155ac73. state=OFFLINE, ts=1369432835898, server=null}, server=kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:35,952 DEBUG [pool-1-thread-1-EventThread] master.OfflineCallback$ExistCallback(106): rs={test,jjj,1369432820320.eb38c552d2a930aa1ce862202da80908. state=OFFLINE, ts=1369432835898, server=null}, server=kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:35,952 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:35,952 DEBUG [pool-1-thread-1-EventThread] master.OfflineCallback$ExistCallback(106): rs={test,kkk,1369432820320.1ce9aafaaa422cef4ce0e1895111c733. state=OFFLINE, ts=1369432835899, server=null}, server=kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:35,952 DEBUG [pool-1-thread-1-EventThread] master.OfflineCallback$ExistCallback(106): rs={test,mmm,1369432820320.ab7b9b9773243cff0ac09f3c379e33dc. state=OFFLINE, ts=1369432835900, server=null}, server=kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:35,952 DEBUG [pool-1-thread-1-EventThread] master.OfflineCallback$ExistCallback(106): rs={test,nnn,1369432820320.53e3aef4cc8815dd31923659bff5ca21. state=OFFLINE, ts=1369432835900, server=null}, server=kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:35,953 DEBUG [pool-1-thread-1-EventThread] master.OfflineCallback$ExistCallback(106): rs={test,ooo,1369432820320.0d6df81a73ad3b2a79add80716da62ac. state=OFFLINE, ts=1369432835901, server=null}, server=kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:35,953 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.AssignmentManager(1512): kiyo.gq1.ygridcore.net,42690,1369432806911 unassigned znodes=7 of total=13 2013-05-24 22:00:35,953 DEBUG [pool-1-thread-1-EventThread] master.OfflineCallback$ExistCallback(106): rs={test,qqq,1369432820320.ec0b9e20bb8d4dc8fee9b3e80c5cae32. state=OFFLINE, ts=1369432835902, server=null}, server=kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:35,953 DEBUG [pool-1-thread-1-EventThread] master.OfflineCallback$ExistCallback(106): rs={test,sss,1369432820320.872960a61a4ceeb9f64577d8b7cb0ddf. state=OFFLINE, ts=1369432835903, server=null}, server=kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:35,954 DEBUG [pool-1-thread-1-EventThread] master.OfflineCallback$ExistCallback(106): rs={test,uuu,1369432820320.dba8b67ade91438cad8a5209736330e6. state=OFFLINE, ts=1369432835903, server=null}, server=kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:35,954 DEBUG [pool-1-thread-1-EventThread] master.OfflineCallback$ExistCallback(106): rs={test,yyy,1369432820321.acf78311e603cb0d02309ae08763d196. state=OFFLINE, ts=1369432835904, server=null}, server=kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:35,954 DEBUG [pool-1-thread-1-EventThread] master.OfflineCallback$ExistCallback(106): rs={test,zzz,1369432820321.0b4d70d096cb6cbce3376acd563c77dd. state=OFFLINE, ts=1369432835906, server=null}, server=kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:35,955 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(583): Nothing to replicate, sleeping 100 times 4 2013-05-24 22:00:35,958 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.AssignmentManager(1512): kiyo.gq1.ygridcore.net,42690,1369432806911 unassigned znodes=13 of total=13 2013-05-24 22:00:35,958 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.RegionStates(264): Region {NAME => 'test,,1369432820315.a88e8c9e73f832adb3635d6996d9e72d.', STARTKEY => '', ENDKEY => 'bbb', ENCODED => a88e8c9e73f832adb3635d6996d9e72d,} transitioned from {test,,1369432820315.a88e8c9e73f832adb3635d6996d9e72d. state=OFFLINE, ts=1369432835896, server=null} to {test,,1369432820315.a88e8c9e73f832adb3635d6996d9e72d. state=PENDING_OPEN, ts=1369432835958, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:35,959 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.RegionStates(264): Region {NAME => 'test,fff,1369432820319.f852f52d87b0b7acd213f5b49074dde6.', STARTKEY => 'fff', ENDKEY => 'ggg', ENCODED => f852f52d87b0b7acd213f5b49074dde6,} transitioned from {test,fff,1369432820319.f852f52d87b0b7acd213f5b49074dde6. state=OFFLINE, ts=1369432835897, server=null} to {test,fff,1369432820319.f852f52d87b0b7acd213f5b49074dde6. state=PENDING_OPEN, ts=1369432835959, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:35,959 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.RegionStates(264): Region {NAME => 'test,ggg,1369432820319.4b3ff3efde12c5afe1220e81d155ac73.', STARTKEY => 'ggg', ENDKEY => 'hhh', ENCODED => 4b3ff3efde12c5afe1220e81d155ac73,} transitioned from {test,ggg,1369432820319.4b3ff3efde12c5afe1220e81d155ac73. state=OFFLINE, ts=1369432835898, server=null} to {test,ggg,1369432820319.4b3ff3efde12c5afe1220e81d155ac73. state=PENDING_OPEN, ts=1369432835959, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:35,959 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.RegionStates(264): Region {NAME => 'test,jjj,1369432820320.eb38c552d2a930aa1ce862202da80908.', STARTKEY => 'jjj', ENDKEY => 'kkk', ENCODED => eb38c552d2a930aa1ce862202da80908,} transitioned from {test,jjj,1369432820320.eb38c552d2a930aa1ce862202da80908. state=OFFLINE, ts=1369432835899, server=null} to {test,jjj,1369432820320.eb38c552d2a930aa1ce862202da80908. state=PENDING_OPEN, ts=1369432835959, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:35,959 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.RegionStates(264): Region {NAME => 'test,kkk,1369432820320.1ce9aafaaa422cef4ce0e1895111c733.', STARTKEY => 'kkk', ENDKEY => 'lll', ENCODED => 1ce9aafaaa422cef4ce0e1895111c733,} transitioned from {test,kkk,1369432820320.1ce9aafaaa422cef4ce0e1895111c733. state=OFFLINE, ts=1369432835899, server=null} to {test,kkk,1369432820320.1ce9aafaaa422cef4ce0e1895111c733. state=PENDING_OPEN, ts=1369432835959, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:35,959 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.RegionStates(264): Region {NAME => 'test,mmm,1369432820320.ab7b9b9773243cff0ac09f3c379e33dc.', STARTKEY => 'mmm', ENDKEY => 'nnn', ENCODED => ab7b9b9773243cff0ac09f3c379e33dc,} transitioned from {test,mmm,1369432820320.ab7b9b9773243cff0ac09f3c379e33dc. state=OFFLINE, ts=1369432835900, server=null} to {test,mmm,1369432820320.ab7b9b9773243cff0ac09f3c379e33dc. state=PENDING_OPEN, ts=1369432835959, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:35,959 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.RegionStates(264): Region {NAME => 'test,nnn,1369432820320.53e3aef4cc8815dd31923659bff5ca21.', STARTKEY => 'nnn', ENDKEY => 'ooo', ENCODED => 53e3aef4cc8815dd31923659bff5ca21,} transitioned from {test,nnn,1369432820320.53e3aef4cc8815dd31923659bff5ca21. state=OFFLINE, ts=1369432835901, server=null} to {test,nnn,1369432820320.53e3aef4cc8815dd31923659bff5ca21. state=PENDING_OPEN, ts=1369432835959, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:35,959 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.RegionStates(264): Region {NAME => 'test,ooo,1369432820320.0d6df81a73ad3b2a79add80716da62ac.', STARTKEY => 'ooo', ENDKEY => 'ppp', ENCODED => 0d6df81a73ad3b2a79add80716da62ac,} transitioned from {test,ooo,1369432820320.0d6df81a73ad3b2a79add80716da62ac. state=OFFLINE, ts=1369432835901, server=null} to {test,ooo,1369432820320.0d6df81a73ad3b2a79add80716da62ac. state=PENDING_OPEN, ts=1369432835959, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:35,960 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.RegionStates(264): Region {NAME => 'test,qqq,1369432820320.ec0b9e20bb8d4dc8fee9b3e80c5cae32.', STARTKEY => 'qqq', ENDKEY => 'rrr', ENCODED => ec0b9e20bb8d4dc8fee9b3e80c5cae32,} transitioned from {test,qqq,1369432820320.ec0b9e20bb8d4dc8fee9b3e80c5cae32. state=OFFLINE, ts=1369432835902, server=null} to {test,qqq,1369432820320.ec0b9e20bb8d4dc8fee9b3e80c5cae32. state=PENDING_OPEN, ts=1369432835960, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:35,960 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.RegionStates(264): Region {NAME => 'test,sss,1369432820320.872960a61a4ceeb9f64577d8b7cb0ddf.', STARTKEY => 'sss', ENDKEY => 'ttt', ENCODED => 872960a61a4ceeb9f64577d8b7cb0ddf,} transitioned from {test,sss,1369432820320.872960a61a4ceeb9f64577d8b7cb0ddf. state=OFFLINE, ts=1369432835903, server=null} to {test,sss,1369432820320.872960a61a4ceeb9f64577d8b7cb0ddf. state=PENDING_OPEN, ts=1369432835960, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:35,960 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.RegionStates(264): Region {NAME => 'test,uuu,1369432820320.dba8b67ade91438cad8a5209736330e6.', STARTKEY => 'uuu', ENDKEY => 'vvv', ENCODED => dba8b67ade91438cad8a5209736330e6,} transitioned from {test,uuu,1369432820320.dba8b67ade91438cad8a5209736330e6. state=OFFLINE, ts=1369432835903, server=null} to {test,uuu,1369432820320.dba8b67ade91438cad8a5209736330e6. state=PENDING_OPEN, ts=1369432835960, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:35,960 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.RegionStates(264): Region {NAME => 'test,yyy,1369432820321.acf78311e603cb0d02309ae08763d196.', STARTKEY => 'yyy', ENDKEY => 'zzz', ENCODED => acf78311e603cb0d02309ae08763d196,} transitioned from {test,yyy,1369432820321.acf78311e603cb0d02309ae08763d196. state=OFFLINE, ts=1369432835904, server=null} to {test,yyy,1369432820321.acf78311e603cb0d02309ae08763d196. state=PENDING_OPEN, ts=1369432835960, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:35,960 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.RegionStates(264): Region {NAME => 'test,zzz,1369432820321.0b4d70d096cb6cbce3376acd563c77dd.', STARTKEY => 'zzz', ENDKEY => '', ENCODED => 0b4d70d096cb6cbce3376acd563c77dd,} transitioned from {test,zzz,1369432820321.0b4d70d096cb6cbce3376acd563c77dd. state=OFFLINE, ts=1369432835906, server=null} to {test,zzz,1369432820321.0b4d70d096cb6cbce3376acd563c77dd. state=PENDING_OPEN, ts=1369432835960, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:35,962 INFO [PRI IPC Server handler 1 on 42690] regionserver.HRegionServer(3454): Received request to open region: test,,1369432820315.a88e8c9e73f832adb3635d6996d9e72d. on kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:35,968 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] zookeeper.ZKAssign(786): regionserver:42690-0x13ed88e2cdf0006 Attempting to transition node a88e8c9e73f832adb3635d6996d9e72d from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-05-24 22:00:35,968 INFO [PRI IPC Server handler 1 on 42690] regionserver.HRegionServer(3454): Received request to open region: test,fff,1369432820319.f852f52d87b0b7acd213f5b49074dde6. on kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:35,970 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] zookeeper.ZKAssign(786): regionserver:42690-0x13ed88e2cdf0006 Attempting to transition node f852f52d87b0b7acd213f5b49074dde6 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-05-24 22:00:35,970 INFO [PRI IPC Server handler 1 on 42690] regionserver.HRegionServer(3454): Received request to open region: test,ggg,1369432820319.4b3ff3efde12c5afe1220e81d155ac73. on kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:35,998 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/region-in-transition/a88e8c9e73f832adb3635d6996d9e72d 2013-05-24 22:00:35,998 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] zookeeper.ZKAssign(862): regionserver:42690-0x13ed88e2cdf0006 Successfully transitioned node a88e8c9e73f832adb3635d6996d9e72d from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-05-24 22:00:35,999 INFO [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegion(4139): HRegion.openHRegion Region name ==test,,1369432820315.a88e8c9e73f832adb3635d6996d9e72d. 2013-05-24 22:00:35,999 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegion(4141): Opening region: {NAME => 'test,,1369432820315.a88e8c9e73f832adb3635d6996d9e72d.', STARTKEY => '', ENDKEY => 'bbb', ENCODED => a88e8c9e73f832adb3635d6996d9e72d,} 2013-05-24 22:00:35,999 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.MetricsRegionSourceImpl(62): Creating new MetricsRegionSourceImpl for table test a88e8c9e73f832adb3635d6996d9e72d 2013-05-24 22:00:35,999 DEBUG [hbase-am-zkevent-worker-pool-2-thread-20] master.AssignmentManager(752): Handling transition=RS_ZK_REGION_OPENING, server=kiyo.gq1.ygridcore.net,42690,1369432806911, region=a88e8c9e73f832adb3635d6996d9e72d, current state from region state map ={test,,1369432820315.a88e8c9e73f832adb3635d6996d9e72d. state=PENDING_OPEN, ts=1369432835958, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:35,999 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] zookeeper.ZKAssign(786): regionserver:42690-0x13ed88e2cdf0006 Attempting to transition node 4b3ff3efde12c5afe1220e81d155ac73 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-05-24 22:00:36,000 INFO [PRI IPC Server handler 1 on 42690] regionserver.HRegionServer(3454): Received request to open region: test,jjj,1369432820320.eb38c552d2a930aa1ce862202da80908. on kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:35,999 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegion(518): Instantiated test,,1369432820315.a88e8c9e73f832adb3635d6996d9e72d. 2013-05-24 22:00:36,004 INFO [StoreOpenerThread-test,,1369432820315.a88e8c9e73f832adb3635d6996d9e72d.-1] compactions.CompactionConfiguration(85): Compaction configuration size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 604800000, major jitter 0.500000 2013-05-24 22:00:36,006 INFO [StoreOpenerThread-test,,1369432820315.a88e8c9e73f832adb3635d6996d9e72d.-1] compactions.CompactionConfiguration(85): Compaction configuration size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 604800000, major jitter 0.500000 2013-05-24 22:00:36,009 INFO [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegion(608): Onlined test,,1369432820315.a88e8c9e73f832adb3635d6996d9e72d.; next sequenceid=1 2013-05-24 22:00:36,009 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] zookeeper.ZKAssign(598): regionserver:42690-0x13ed88e2cdf0006 Attempting to retransition the opening state of node a88e8c9e73f832adb3635d6996d9e72d 2013-05-24 22:00:36,039 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/region-in-transition/f852f52d87b0b7acd213f5b49074dde6 2013-05-24 22:00:36,040 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] zookeeper.ZKAssign(862): regionserver:42690-0x13ed88e2cdf0006 Successfully transitioned node f852f52d87b0b7acd213f5b49074dde6 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-05-24 22:00:36,040 INFO [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegion(4139): HRegion.openHRegion Region name ==test,fff,1369432820319.f852f52d87b0b7acd213f5b49074dde6. 2013-05-24 22:00:36,040 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegion(4141): Opening region: {NAME => 'test,fff,1369432820319.f852f52d87b0b7acd213f5b49074dde6.', STARTKEY => 'fff', ENDKEY => 'ggg', ENCODED => f852f52d87b0b7acd213f5b49074dde6,} 2013-05-24 22:00:36,041 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.MetricsRegionSourceImpl(62): Creating new MetricsRegionSourceImpl for table test f852f52d87b0b7acd213f5b49074dde6 2013-05-24 22:00:36,041 DEBUG [hbase-am-zkevent-worker-pool-2-thread-3] master.AssignmentManager(752): Handling transition=RS_ZK_REGION_OPENING, server=kiyo.gq1.ygridcore.net,42690,1369432806911, region=f852f52d87b0b7acd213f5b49074dde6, current state from region state map ={test,fff,1369432820319.f852f52d87b0b7acd213f5b49074dde6. state=PENDING_OPEN, ts=1369432835959, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:36,041 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegion(518): Instantiated test,fff,1369432820319.f852f52d87b0b7acd213f5b49074dde6. 2013-05-24 22:00:36,041 INFO [PRI IPC Server handler 1 on 42690] regionserver.HRegionServer(3454): Received request to open region: test,kkk,1369432820320.1ce9aafaaa422cef4ce0e1895111c733. on kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:36,042 INFO [PostOpenDeployTasks:a88e8c9e73f832adb3635d6996d9e72d] regionserver.HRegionServer(1657): Post open deploy tasks for region=test,,1369432820315.a88e8c9e73f832adb3635d6996d9e72d. 2013-05-24 22:00:36,045 INFO [StoreOpenerThread-test,fff,1369432820319.f852f52d87b0b7acd213f5b49074dde6.-1] compactions.CompactionConfiguration(85): Compaction configuration size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 604800000, major jitter 0.500000 2013-05-24 22:00:36,047 INFO [StoreOpenerThread-test,fff,1369432820319.f852f52d87b0b7acd213f5b49074dde6.-1] compactions.CompactionConfiguration(85): Compaction configuration size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 604800000, major jitter 0.500000 2013-05-24 22:00:36,051 INFO [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegion(608): Onlined test,fff,1369432820319.f852f52d87b0b7acd213f5b49074dde6.; next sequenceid=1 2013-05-24 22:00:36,051 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] zookeeper.ZKAssign(598): regionserver:42690-0x13ed88e2cdf0006 Attempting to retransition the opening state of node f852f52d87b0b7acd213f5b49074dde6 2013-05-24 22:00:36,114 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/region-in-transition/4b3ff3efde12c5afe1220e81d155ac73 2013-05-24 22:00:36,114 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] zookeeper.ZKAssign(862): regionserver:42690-0x13ed88e2cdf0006 Successfully transitioned node 4b3ff3efde12c5afe1220e81d155ac73 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-05-24 22:00:36,115 INFO [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(4139): HRegion.openHRegion Region name ==test,ggg,1369432820319.4b3ff3efde12c5afe1220e81d155ac73. 2013-05-24 22:00:36,115 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(4141): Opening region: {NAME => 'test,ggg,1369432820319.4b3ff3efde12c5afe1220e81d155ac73.', STARTKEY => 'ggg', ENDKEY => 'hhh', ENCODED => 4b3ff3efde12c5afe1220e81d155ac73,} 2013-05-24 22:00:36,115 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.MetricsRegionSourceImpl(62): Creating new MetricsRegionSourceImpl for table test 4b3ff3efde12c5afe1220e81d155ac73 2013-05-24 22:00:36,115 DEBUG [hbase-am-zkevent-worker-pool-2-thread-7] master.AssignmentManager(752): Handling transition=RS_ZK_REGION_OPENING, server=kiyo.gq1.ygridcore.net,42690,1369432806911, region=4b3ff3efde12c5afe1220e81d155ac73, current state from region state map ={test,ggg,1369432820319.4b3ff3efde12c5afe1220e81d155ac73. state=PENDING_OPEN, ts=1369432835959, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:36,116 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(518): Instantiated test,ggg,1369432820319.4b3ff3efde12c5afe1220e81d155ac73. 2013-05-24 22:00:36,116 INFO [PRI IPC Server handler 1 on 42690] regionserver.HRegionServer(3454): Received request to open region: test,mmm,1369432820320.ab7b9b9773243cff0ac09f3c379e33dc. on kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:36,119 INFO [PostOpenDeployTasks:f852f52d87b0b7acd213f5b49074dde6] regionserver.HRegionServer(1657): Post open deploy tasks for region=test,fff,1369432820319.f852f52d87b0b7acd213f5b49074dde6. 2013-05-24 22:00:36,119 INFO [PRI IPC Server handler 1 on 42690] regionserver.HRegionServer(3454): Received request to open region: test,nnn,1369432820320.53e3aef4cc8815dd31923659bff5ca21. on kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:36,119 DEBUG [Thread-584] client.MetaScanner(160): Scanning .META. starting at row=test,sss,00000000000000 for max=10 with caching=10 2013-05-24 22:00:36,119 INFO [StoreOpenerThread-test,ggg,1369432820319.4b3ff3efde12c5afe1220e81d155ac73.-1] compactions.CompactionConfiguration(85): Compaction configuration size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 604800000, major jitter 0.500000 2013-05-24 22:00:36,120 DEBUG [Thread-584] client.ClientScanner(96): Creating scanner over .META. starting at key 'test,sss,00000000000000' 2013-05-24 22:00:36,120 DEBUG [Thread-584] client.ClientScanner(209): Advancing internal scanner to startKey at 'test,sss,00000000000000' 2013-05-24 22:00:36,122 INFO [PRI IPC Server handler 1 on 42690] regionserver.HRegionServer(3454): Received request to open region: test,ooo,1369432820320.0d6df81a73ad3b2a79add80716da62ac. on kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:36,123 INFO [StoreOpenerThread-test,ggg,1369432820319.4b3ff3efde12c5afe1220e81d155ac73.-1] compactions.CompactionConfiguration(85): Compaction configuration size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 604800000, major jitter 0.500000 2013-05-24 22:00:36,124 DEBUG [Thread-584] client.ClientScanner(196): Finished scanning region {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,} 2013-05-24 22:00:36,124 INFO [PRI IPC Server handler 1 on 42690] regionserver.HRegionServer(3454): Received request to open region: test,qqq,1369432820320.ec0b9e20bb8d4dc8fee9b3e80c5cae32. on kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:36,124 DEBUG [Thread-584] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,sss,1369432820320.872960a61a4ceeb9f64577d8b7cb0ddf. is kiyo.gq1.ygridcore.net:49701 2013-05-24 22:00:36,126 INFO [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(608): Onlined test,ggg,1369432820319.4b3ff3efde12c5afe1220e81d155ac73.; next sequenceid=1 2013-05-24 22:00:36,127 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] zookeeper.ZKAssign(598): regionserver:42690-0x13ed88e2cdf0006 Attempting to retransition the opening state of node 4b3ff3efde12c5afe1220e81d155ac73 2013-05-24 22:00:36,263 INFO [Thread-586-EventThread] hbase.HBaseTestingUtility$1(1837): Monitor ZKW received event=WatchedEvent state:SyncConnected type:None path:null 2013-05-24 22:00:36,281 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:42690-0x13ed88e2cdf0006 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/recovering-regions/a88e8c9e73f832adb3635d6996d9e72d 2013-05-24 22:00:36,282 INFO [PRI IPC Server handler 1 on 42690] regionserver.HRegionServer(3454): Received request to open region: test,sss,1369432820320.872960a61a4ceeb9f64577d8b7cb0ddf. on kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:36,284 INFO [PostOpenDeployTasks:4b3ff3efde12c5afe1220e81d155ac73] regionserver.HRegionServer(1657): Post open deploy tasks for region=test,ggg,1369432820319.4b3ff3efde12c5afe1220e81d155ac73. 2013-05-24 22:00:36,357 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432834215 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:36,362 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:36,362 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,39146,1369432814418-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:39146-0x13ed88e2cdf0011 Received ZooKeeper Event, type=None, state=Disconnected, path=null 2013-05-24 22:00:36,362 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,39146,1369432814418-EventThread] zookeeper.ZooKeeperWatcher(389): regionserver:39146-0x13ed88e2cdf0011 Received Disconnected from ZooKeeper, ignoring 2013-05-24 22:00:36,363 INFO [Thread-586-EventThread] hbase.HBaseTestingUtility$1(1837): Monitor ZKW received event=WatchedEvent state:Disconnected type:None path:null 2013-05-24 22:00:36,365 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(583): Nothing to replicate, sleeping 100 times 5 2013-05-24 22:00:36,412 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:49701-0x13ed88e2cdf0005 Received ZooKeeper Event, type=None, state=Expired, path=null 2013-05-24 22:00:36,413 FATAL [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997-EventThread] regionserver.HRegionServer(1705): ABORTING region server kiyo.gq1.ygridcore.net,49701,1369432806997: regionserver:49701-0x13ed88e2cdf0005 regionserver:49701-0x13ed88e2cdf0005 received expired from ZooKeeper, aborting org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:398) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:316) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495) 2013-05-24 22:00:36,413 FATAL [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997-EventThread] regionserver.HRegionServer(1713): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2013-05-24 22:00:36,413 INFO [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997-EventThread] regionserver.HRegionServer(1639): STOPPED: regionserver:49701-0x13ed88e2cdf0005 regionserver:49701-0x13ed88e2cdf0005 received expired from ZooKeeper, aborting 2013-05-24 22:00:36,455 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:51523-0x13ed88e2cdf0010 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/2/rs/kiyo.gq1.ygridcore.net,39146,1369432814418 2013-05-24 22:00:36,455 INFO [Thread-586] hbase.HBaseTestingUtility(1853): ZK Closed Session 0x13ed88e2cdf0011 2013-05-24 22:00:36,457 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:53632-0x13ed88e2cdf000f Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/2/rs/kiyo.gq1.ygridcore.net,39146,1369432814418 2013-05-24 22:00:36,457 INFO [pool-1-thread-1-EventThread] zookeeper.RegionServerTracker(94): RegionServer ephemeral node deleted, processing expiration [kiyo.gq1.ygridcore.net,39146,1369432814418] 2013-05-24 22:00:36,513 WARN [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] zookeeper.RecoverableZooKeeper(237): Possibly transient ZooKeeper, quorum=localhost:51411, exception=org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /1/replication/rs/kiyo.gq1.ygridcore.net,49701,1369432806997/2/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432832342 2013-05-24 22:00:36,513 ERROR [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] zookeeper.RecoverableZooKeeper(239): ZooKeeper setData failed after 1 retries 2013-05-24 22:00:36,513 FATAL [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] regionserver.HRegionServer(1705): ABORTING region server kiyo.gq1.ygridcore.net,49701,1369432806997: Failed to write replication hlog position (filename=kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432832342, position=44499) org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /1/replication/rs/kiyo.gq1.ygridcore.net,49701,1369432806997/2/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432832342 at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1266) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.setData(RecoverableZooKeeper.java:368) at org.apache.hadoop.hbase.zookeeper.ZKUtil.setData(ZKUtil.java:884) at org.apache.hadoop.hbase.zookeeper.ZKUtil.setData(ZKUtil.java:934) at org.apache.hadoop.hbase.zookeeper.ZKUtil.setData(ZKUtil.java:928) at org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl.setLogPosition(ReplicationQueuesZKImpl.java:117) at org.apache.hadoop.hbase.replication.ReplicationZookeeper.writeReplicationStatus(ReplicationZookeeper.java:274) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.logPositionAndCleanOldLogs(ReplicationSourceManager.java:159) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.shipEdits(ReplicationSource.java:649) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:378) 2013-05-24 22:00:36,514 FATAL [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] regionserver.HRegionServer(1713): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2013-05-24 22:00:36,514 INFO [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] regionserver.HRegionServer(1639): STOPPED: Failed to write replication hlog position (filename=kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432832342, position=44499) 2013-05-24 22:00:36,514 INFO [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] client.HConnectionManager$HConnectionImplementation(1640): Closing zookeeper sessionid=0x13ed88e2cdf0009 2013-05-24 22:00:36,564 DEBUG [PostOpenDeployTasks:a88e8c9e73f832adb3635d6996d9e72d] regionserver.HRegionServer(4268): Update last flushed sequence id of region a88e8c9e73f832adb3635d6996d9e72d for kiyo.gq1.ygridcore.net,49701,1369432806997 2013-05-24 22:00:36,564 DEBUG [pool-1-thread-1-EventThread] master.AssignmentManager(2991): based on AM, current region=.META.,,1.1028785192 is on server=kiyo.gq1.ygridcore.net,51523,1369432814392 server being checked: kiyo.gq1.ygridcore.net,39146,1369432814418 2013-05-24 22:00:36,564 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:42690-0x13ed88e2cdf0006 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/recovering-regions/f852f52d87b0b7acd213f5b49074dde6 2013-05-24 22:00:36,565 DEBUG [pool-1-thread-1-EventThread] master.ServerManager(511): Added=kiyo.gq1.ygridcore.net,39146,1369432814418 to dead servers, submitted shutdown handler to be executed meta=false 2013-05-24 22:00:36,565 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:53632-0x13ed88e2cdf000f Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/2/rs 2013-05-24 22:00:36,566 INFO [PRI IPC Server handler 1 on 42690] regionserver.HRegionServer(3454): Received request to open region: test,uuu,1369432820320.dba8b67ade91438cad8a5209736330e6. on kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:36,607 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997.replicationSource,2] regionserver.ReplicationSource(388): Source exiting 2 2013-05-24 22:00:36,608 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392-EventThread] zookeeper.ZKUtil(431): regionserver:51523-0x13ed88e2cdf0010 Set watcher on existing znode=/2/rs/kiyo.gq1.ygridcore.net,51523,1369432814392 2013-05-24 22:00:36,608 INFO [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997] client.HConnectionManager$HConnectionImplementation(1640): Closing zookeeper sessionid=0x13ed88e2cdf000d 2013-05-24 22:00:36,608 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392-EventThread] regionserver.ReplicationSourceManager$OtherRegionServerWatcher(467): /2/rs/kiyo.gq1.ygridcore.net,39146,1369432814418 znode expired, trying to lock it 2013-05-24 22:00:36,609 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:51523-0x13ed88e2cdf0010 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/2/rs 2013-05-24 22:00:36,753 DEBUG [PostOpenDeployTasks:f852f52d87b0b7acd213f5b49074dde6] regionserver.HRegionServer(4268): Update last flushed sequence id of region f852f52d87b0b7acd213f5b49074dde6 for kiyo.gq1.ygridcore.net,49701,1369432806997 2013-05-24 22:00:36,754 INFO [PRI IPC Server handler 1 on 42690] regionserver.HRegionServer(3454): Received request to open region: test,yyy,1369432820321.acf78311e603cb0d02309ae08763d196. on kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:36,832 WARN [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997] zookeeper.RecoverableZooKeeper(237): Possibly transient ZooKeeper, quorum=localhost:51411, exception=org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /1/rs/kiyo.gq1.ygridcore.net,49701,1369432806997 2013-05-24 22:00:36,832 INFO [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997] util.RetryCounter(54): Sleeping 20ms before retry #1... 2013-05-24 22:00:36,835 INFO [PRI IPC Server handler 1 on 42690] regionserver.HRegionServer(3454): Received request to open region: test,zzz,1369432820321.0b4d70d096cb6cbce3376acd563c77dd. on kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:36,836 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(431): master:53632-0x13ed88e2cdf000f Set watcher on existing znode=/2/rs/kiyo.gq1.ygridcore.net,51523,1369432814392 2013-05-24 22:00:36,842 INFO [PostOpenDeployTasks:a88e8c9e73f832adb3635d6996d9e72d] catalog.MetaEditor(432): Updated row test,,1369432820315.a88e8c9e73f832adb3635d6996d9e72d. with server=kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:36,842 INFO [PostOpenDeployTasks:a88e8c9e73f832adb3635d6996d9e72d] regionserver.HRegionServer(1682): Done with post open deploy task for region=test,,1369432820315.a88e8c9e73f832adb3635d6996d9e72d. 2013-05-24 22:00:36,842 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] zookeeper.ZKAssign(786): regionserver:42690-0x13ed88e2cdf0006 Attempting to transition node a88e8c9e73f832adb3635d6996d9e72d from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-05-24 22:00:36,850 INFO [ReplicationExecutor-0] replication.ReplicationQueuesZKImpl(152): Moving kiyo.gq1.ygridcore.net,49701,1369432806997's hlogs to my queue 2013-05-24 22:00:36,853 WARN [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997] zookeeper.RecoverableZooKeeper(237): Possibly transient ZooKeeper, quorum=localhost:51411, exception=org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /1/rs/kiyo.gq1.ygridcore.net,49701,1369432806997 2013-05-24 22:00:36,853 ERROR [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997] zookeeper.RecoverableZooKeeper(239): ZooKeeper delete failed after 1 retries 2013-05-24 22:00:36,853 WARN [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.HRegionServer(935): Failed deleting my ephemeral node org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /1/rs/kiyo.gq1.ygridcore.net,49701,1369432806997 at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:152) at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1268) at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1257) at org.apache.hadoop.hbase.regionserver.HRegionServer.deleteMyEphemeralNode(HRegionServer.java:1193) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:933) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:158) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:110) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:142) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:337) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1457) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.util.Methods.call(Methods.java:41) at org.apache.hadoop.hbase.security.User.call(User.java:420) at org.apache.hadoop.hbase.security.User.access$300(User.java:51) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:260) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:140) at java.lang.Thread.run(Thread.java:662) 2013-05-24 22:00:36,854 INFO [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.HRegionServer(941): stopping server kiyo.gq1.ygridcore.net,49701,1369432806997; zookeeper connection closed. 2013-05-24 22:00:36,854 INFO [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.HRegionServer(944): RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997 exiting 2013-05-24 22:00:36,855 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@3d949a87] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(193): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@3d949a87 2013-05-24 22:00:36,868 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432834215 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:36,871 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:36,875 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(583): Nothing to replicate, sleeping 100 times 6 2013-05-24 22:00:37,007 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:42690-0x13ed88e2cdf0006 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/recovering-regions/4b3ff3efde12c5afe1220e81d155ac73 2013-05-24 22:00:37,008 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392-EventThread] zookeeper.ZKUtil(431): regionserver:51523-0x13ed88e2cdf0010 Set watcher on existing znode=/2/rs/kiyo.gq1.ygridcore.net,51523,1369432814392 2013-05-24 22:00:37,008 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,53632,1369432814258-0] client.ClientScanner(96): Creating scanner over .META. starting at key '' 2013-05-24 22:00:37,008 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,53632,1369432814258-0] client.ClientScanner(209): Advancing internal scanner to startKey at '' 2013-05-24 22:00:37,009 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.AssignmentManager(1634): Bulk assigning done for kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:37,009 INFO [hbase-am-zkevent-worker-pool-2-thread-20] master.RegionStates(264): Region {NAME => 'test,,1369432820315.a88e8c9e73f832adb3635d6996d9e72d.', STARTKEY => '', ENDKEY => 'bbb', ENCODED => a88e8c9e73f832adb3635d6996d9e72d,} transitioned from {test,,1369432820315.a88e8c9e73f832adb3635d6996d9e72d. state=PENDING_OPEN, ts=1369432835958, server=kiyo.gq1.ygridcore.net,42690,1369432806911} to {test,,1369432820315.a88e8c9e73f832adb3635d6996d9e72d. state=OPENING, ts=1369432837009, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:37,010 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.AssignmentManager(2753): Waiting on null to clear regions-in-transition 2013-05-24 22:00:37,010 INFO [hbase-am-zkevent-worker-pool-2-thread-7] master.RegionStates(264): Region {NAME => 'test,ggg,1369432820319.4b3ff3efde12c5afe1220e81d155ac73.', STARTKEY => 'ggg', ENDKEY => 'hhh', ENCODED => 4b3ff3efde12c5afe1220e81d155ac73,} transitioned from {test,ggg,1369432820319.4b3ff3efde12c5afe1220e81d155ac73. state=PENDING_OPEN, ts=1369432835959, server=kiyo.gq1.ygridcore.net,42690,1369432806911} to {test,ggg,1369432820319.4b3ff3efde12c5afe1220e81d155ac73. state=OPENING, ts=1369432837010, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:37,010 INFO [PostOpenDeployTasks:f852f52d87b0b7acd213f5b49074dde6] catalog.MetaEditor(432): Updated row test,fff,1369432820319.f852f52d87b0b7acd213f5b49074dde6. with server=kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:37,010 INFO [hbase-am-zkevent-worker-pool-2-thread-3] master.RegionStates(264): Region {NAME => 'test,fff,1369432820319.f852f52d87b0b7acd213f5b49074dde6.', STARTKEY => 'fff', ENDKEY => 'ggg', ENCODED => f852f52d87b0b7acd213f5b49074dde6,} transitioned from {test,fff,1369432820319.f852f52d87b0b7acd213f5b49074dde6. state=PENDING_OPEN, ts=1369432835959, server=kiyo.gq1.ygridcore.net,42690,1369432806911} to {test,fff,1369432820319.f852f52d87b0b7acd213f5b49074dde6. state=OPENING, ts=1369432837010, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:37,010 INFO [PostOpenDeployTasks:f852f52d87b0b7acd213f5b49074dde6] regionserver.HRegionServer(1682): Done with post open deploy task for region=test,fff,1369432820319.f852f52d87b0b7acd213f5b49074dde6. 2013-05-24 22:00:37,010 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] zookeeper.ZKAssign(786): regionserver:42690-0x13ed88e2cdf0006 Attempting to transition node f852f52d87b0b7acd213f5b49074dde6 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-05-24 22:00:37,126 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,53632,1369432814258-0] client.ClientScanner(196): Finished scanning region {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,} 2013-05-24 22:00:37,127 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,53632,1369432814258-0] handler.ServerShutdownHandler(184): Splitting logs for kiyo.gq1.ygridcore.net,39146,1369432814418 before assignment. 2013-05-24 22:00:37,168 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/region-in-transition/a88e8c9e73f832adb3635d6996d9e72d 2013-05-24 22:00:37,168 DEBUG [PostOpenDeployTasks:4b3ff3efde12c5afe1220e81d155ac73] regionserver.HRegionServer(4268): Update last flushed sequence id of region 4b3ff3efde12c5afe1220e81d155ac73 for kiyo.gq1.ygridcore.net,49701,1369432806997 2013-05-24 22:00:37,169 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] zookeeper.ZKAssign(862): regionserver:42690-0x13ed88e2cdf0006 Successfully transitioned node a88e8c9e73f832adb3635d6996d9e72d from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-05-24 22:00:37,169 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] handler.OpenRegionHandler(373): region transitioned to opened in zookeeper: {NAME => 'test,,1369432820315.a88e8c9e73f832adb3635d6996d9e72d.', STARTKEY => '', ENDKEY => 'bbb', ENCODED => a88e8c9e73f832adb3635d6996d9e72d,}, server: kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:37,169 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] handler.OpenRegionHandler(186): Opened test,,1369432820315.a88e8c9e73f832adb3635d6996d9e72d. on server:kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:37,169 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] zookeeper.ZKAssign(786): regionserver:42690-0x13ed88e2cdf0006 Attempting to transition node eb38c552d2a930aa1ce862202da80908 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-05-24 22:00:37,171 DEBUG [hbase-am-zkevent-worker-pool-2-thread-2] master.AssignmentManager(752): Handling transition=RS_ZK_REGION_OPENED, server=kiyo.gq1.ygridcore.net,42690,1369432806911, region=a88e8c9e73f832adb3635d6996d9e72d, current state from region state map ={test,,1369432820315.a88e8c9e73f832adb3635d6996d9e72d. state=OPENING, ts=1369432837009, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:37,171 INFO [hbase-am-zkevent-worker-pool-2-thread-2] master.RegionStates(264): Region {NAME => 'test,,1369432820315.a88e8c9e73f832adb3635d6996d9e72d.', STARTKEY => '', ENDKEY => 'bbb', ENCODED => a88e8c9e73f832adb3635d6996d9e72d,} transitioned from {test,,1369432820315.a88e8c9e73f832adb3635d6996d9e72d. state=OPENING, ts=1369432837009, server=kiyo.gq1.ygridcore.net,42690,1369432806911} to {test,,1369432820315.a88e8c9e73f832adb3635d6996d9e72d. state=OPEN, ts=1369432837171, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:37,172 DEBUG [MASTER_OPEN_REGION-kiyo.gq1.ygridcore.net,49101,1369432806564-2] handler.OpenedRegionHandler(146): Handling OPENED event for test,,1369432820315.a88e8c9e73f832adb3635d6996d9e72d. from kiyo.gq1.ygridcore.net,42690,1369432806911; deleting unassigned node 2013-05-24 22:00:37,172 DEBUG [MASTER_OPEN_REGION-kiyo.gq1.ygridcore.net,49101,1369432806564-2] zookeeper.ZKAssign(405): master:49101-0x13ed88e2cdf0004 Deleting existing unassigned node for a88e8c9e73f832adb3635d6996d9e72d that is in expected state RS_ZK_REGION_OPENED 2013-05-24 22:00:37,216 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/region-in-transition/f852f52d87b0b7acd213f5b49074dde6 2013-05-24 22:00:37,217 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] zookeeper.ZKAssign(862): regionserver:42690-0x13ed88e2cdf0006 Successfully transitioned node f852f52d87b0b7acd213f5b49074dde6 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-05-24 22:00:37,217 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,53632,1369432814258-0] client.ClientScanner(96): Creating scanner over .META. starting at key '' 2013-05-24 22:00:37,217 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] handler.OpenRegionHandler(373): region transitioned to opened in zookeeper: {NAME => 'test,fff,1369432820319.f852f52d87b0b7acd213f5b49074dde6.', STARTKEY => 'fff', ENDKEY => 'ggg', ENCODED => f852f52d87b0b7acd213f5b49074dde6,}, server: kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:37,217 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,53632,1369432814258-0] client.ClientScanner(209): Advancing internal scanner to startKey at '' 2013-05-24 22:00:37,218 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] handler.OpenRegionHandler(186): Opened test,fff,1369432820319.f852f52d87b0b7acd213f5b49074dde6. on server:kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:37,218 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] zookeeper.ZKAssign(786): regionserver:42690-0x13ed88e2cdf0006 Attempting to transition node 1ce9aafaaa422cef4ce0e1895111c733 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-05-24 22:00:37,219 INFO [PostOpenDeployTasks:4b3ff3efde12c5afe1220e81d155ac73] catalog.MetaEditor(432): Updated row test,ggg,1369432820319.4b3ff3efde12c5afe1220e81d155ac73. with server=kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:37,219 INFO [PostOpenDeployTasks:4b3ff3efde12c5afe1220e81d155ac73] regionserver.HRegionServer(1682): Done with post open deploy task for region=test,ggg,1369432820319.4b3ff3efde12c5afe1220e81d155ac73. 2013-05-24 22:00:37,219 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] zookeeper.ZKAssign(786): regionserver:42690-0x13ed88e2cdf0006 Attempting to transition node 4b3ff3efde12c5afe1220e81d155ac73 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-05-24 22:00:37,361 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/region-in-transition/eb38c552d2a930aa1ce862202da80908 2013-05-24 22:00:37,361 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] zookeeper.ZKAssign(862): regionserver:42690-0x13ed88e2cdf0006 Successfully transitioned node eb38c552d2a930aa1ce862202da80908 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-05-24 22:00:37,362 INFO [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegion(4139): HRegion.openHRegion Region name ==test,jjj,1369432820320.eb38c552d2a930aa1ce862202da80908. 2013-05-24 22:00:37,362 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegion(4141): Opening region: {NAME => 'test,jjj,1369432820320.eb38c552d2a930aa1ce862202da80908.', STARTKEY => 'jjj', ENDKEY => 'kkk', ENCODED => eb38c552d2a930aa1ce862202da80908,} 2013-05-24 22:00:37,362 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.MetricsRegionSourceImpl(62): Creating new MetricsRegionSourceImpl for table test eb38c552d2a930aa1ce862202da80908 2013-05-24 22:00:37,363 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegion(518): Instantiated test,jjj,1369432820320.eb38c552d2a930aa1ce862202da80908. 2013-05-24 22:00:37,363 DEBUG [hbase-am-zkevent-worker-pool-2-thread-5] master.AssignmentManager(752): Handling transition=RS_ZK_REGION_OPENED, server=kiyo.gq1.ygridcore.net,42690,1369432806911, region=f852f52d87b0b7acd213f5b49074dde6, current state from region state map ={test,fff,1369432820319.f852f52d87b0b7acd213f5b49074dde6. state=OPENING, ts=1369432837010, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:37,363 INFO [hbase-am-zkevent-worker-pool-2-thread-5] master.RegionStates(264): Region {NAME => 'test,fff,1369432820319.f852f52d87b0b7acd213f5b49074dde6.', STARTKEY => 'fff', ENDKEY => 'ggg', ENCODED => f852f52d87b0b7acd213f5b49074dde6,} transitioned from {test,fff,1369432820319.f852f52d87b0b7acd213f5b49074dde6. state=OPENING, ts=1369432837010, server=kiyo.gq1.ygridcore.net,42690,1369432806911} to {test,fff,1369432820319.f852f52d87b0b7acd213f5b49074dde6. state=OPEN, ts=1369432837363, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:37,363 DEBUG [MASTER_OPEN_REGION-kiyo.gq1.ygridcore.net,49101,1369432806564-3] handler.OpenedRegionHandler(146): Handling OPENED event for test,fff,1369432820319.f852f52d87b0b7acd213f5b49074dde6. from kiyo.gq1.ygridcore.net,42690,1369432806911; deleting unassigned node 2013-05-24 22:00:37,364 DEBUG [MASTER_OPEN_REGION-kiyo.gq1.ygridcore.net,49101,1369432806564-3] zookeeper.ZKAssign(405): master:49101-0x13ed88e2cdf0004 Deleting existing unassigned node for f852f52d87b0b7acd213f5b49074dde6 that is in expected state RS_ZK_REGION_OPENED 2013-05-24 22:00:37,367 INFO [StoreOpenerThread-test,jjj,1369432820320.eb38c552d2a930aa1ce862202da80908.-1] compactions.CompactionConfiguration(85): Compaction configuration size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 604800000, major jitter 0.500000 2013-05-24 22:00:37,370 INFO [StoreOpenerThread-test,jjj,1369432820320.eb38c552d2a930aa1ce862202da80908.-1] compactions.CompactionConfiguration(85): Compaction configuration size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 604800000, major jitter 0.500000 2013-05-24 22:00:37,374 INFO [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegion(608): Onlined test,jjj,1369432820320.eb38c552d2a930aa1ce862202da80908.; next sequenceid=1 2013-05-24 22:00:37,374 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] zookeeper.ZKAssign(598): regionserver:42690-0x13ed88e2cdf0006 Attempting to retransition the opening state of node eb38c552d2a930aa1ce862202da80908 2013-05-24 22:00:37,394 DEBUG [IPC Server handler 4 on 53632] master.ServerManager(337): Server REPORT rejected; currently processing kiyo.gq1.ygridcore.net,39146,1369432814418 as dead server 2013-05-24 22:00:37,396 FATAL [RegionServer:1;kiyo.gq1.ygridcore.net,39146,1369432814418] regionserver.HRegionServer(1705): ABORTING region server kiyo.gq1.ygridcore.net,39146,1369432814418: org.apache.hadoop.hbase.exceptions.YouAreDeadException: Server REPORT rejected; currently processing kiyo.gq1.ygridcore.net,39146,1369432814418 as dead server at org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:338) at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:253) at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:1227) at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:3800) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2103) at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1810) org.apache.hadoop.hbase.exceptions.YouAreDeadException: org.apache.hadoop.hbase.exceptions.YouAreDeadException: Server REPORT rejected; currently processing kiyo.gq1.ygridcore.net,39146,1369432814418 as dead server at org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:338) at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:253) at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:1227) at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:3800) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2103) at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1810) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:227) at org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:974) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:821) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:158) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:110) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:142) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:337) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1457) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.util.Methods.call(Methods.java:41) at org.apache.hadoop.hbase.security.User.call(User.java:420) at org.apache.hadoop.hbase.security.User.access$300(User.java:51) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:260) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:140) at java.lang.Thread.run(Thread.java:662) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.exceptions.YouAreDeadException): org.apache.hadoop.hbase.exceptions.YouAreDeadException: Server REPORT rejected; currently processing kiyo.gq1.ygridcore.net,39146,1369432814418 as dead server at org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:338) at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:253) at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:1227) at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:3800) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2103) at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1810) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1335) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1539) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1596) at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerReport(RegionServerStatusProtos.java:4095) at org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:972) ... 17 more 2013-05-24 22:00:37,397 FATAL [RegionServer:1;kiyo.gq1.ygridcore.net,39146,1369432814418] regionserver.HRegionServer(1713): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2013-05-24 22:00:37,397 ERROR [IPC Server handler 0 on 53632] master.HMaster(1246): Region server kiyo.gq1.ygridcore.net,39146,1369432814418 reported a fatal error: ABORTING region server kiyo.gq1.ygridcore.net,39146,1369432814418: org.apache.hadoop.hbase.exceptions.YouAreDeadException: Server REPORT rejected; currently processing kiyo.gq1.ygridcore.net,39146,1369432814418 as dead server at org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:338) at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:253) at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:1227) at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:3800) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2103) at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1810) Cause: org.apache.hadoop.hbase.exceptions.YouAreDeadException: org.apache.hadoop.hbase.exceptions.YouAreDeadException: Server REPORT rejected; currently processing kiyo.gq1.ygridcore.net,39146,1369432814418 as dead server at org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:338) at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:253) at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:1227) at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:3800) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2103) at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1810) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:227) at org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:974) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:821) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:158) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:110) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:142) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:337) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1457) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.util.Methods.call(Methods.java:41) at org.apache.hadoop.hbase.security.User.call(User.java:420) at org.apache.hadoop.hbase.security.User.access$300(User.java:51) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:260) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:140) at java.lang.Thread.run(Thread.java:662) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.exceptions.YouAreDeadException): org.apache.hadoop.hbase.exceptions.YouAreDeadException: Server REPORT rejected; currently processing kiyo.gq1.ygridcore.net,39146,1369432814418 as dead server at org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:338) at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:253) at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:1227) at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:3800) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2103) at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1810) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1335) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1539) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1596) at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerReport(RegionServerStatusProtos.java:4095) at org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:972) ... 17 more 2013-05-24 22:00:37,397 INFO [RegionServer:1;kiyo.gq1.ygridcore.net,39146,1369432814418] regionserver.HRegionServer(1639): STOPPED: org.apache.hadoop.hbase.exceptions.YouAreDeadException: Server REPORT rejected; currently processing kiyo.gq1.ygridcore.net,39146,1369432814418 as dead server at org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:338) at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:253) at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:1227) at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:3800) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2103) at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1810) 2013-05-24 22:00:37,400 INFO [RegionServer:1;kiyo.gq1.ygridcore.net,39146,1369432814418] regionserver.SplitLogWorker(585): Sending interrupt to stop the worker thread 2013-05-24 22:00:37,400 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,39146,1369432814418] regionserver.SplitLogWorker(267): SplitLogWorker interrupted while waiting for task, exiting: java.lang.InterruptedException 2013-05-24 22:00:37,400 INFO [Thread-341] regionserver.MemStoreFlusher$FlushHandler(267): Thread-341 exiting 2013-05-24 22:00:37,400 INFO [RegionServer:1;kiyo.gq1.ygridcore.net,39146,1369432814418] snapshot.RegionServerSnapshotManager(151): Stopping RegionServerSnapshotManager abruptly. 2013-05-24 22:00:37,401 INFO [RegionServer:1;kiyo.gq1.ygridcore.net,39146,1369432814418.compactionChecker] hbase.Chore(93): RegionServer:1;kiyo.gq1.ygridcore.net,39146,1369432814418.compactionChecker exiting 2013-05-24 22:00:37,400 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,39146,1369432814418] regionserver.SplitLogWorker(199): SplitLogWorker kiyo.gq1.ygridcore.net,39146,1369432814418 exiting 2013-05-24 22:00:37,400 INFO [RegionServer:1;kiyo.gq1.ygridcore.net,39146,1369432814418.logRoller] regionserver.LogRoller(119): LogRoller exiting. 2013-05-24 22:00:37,402 INFO [RegionServer:1;kiyo.gq1.ygridcore.net,39146,1369432814418] regionserver.HRegionServer(882): aborting server kiyo.gq1.ygridcore.net,39146,1369432814418 2013-05-24 22:00:37,402 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,39146,1369432814418] catalog.CatalogTracker(208): Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@6a4ef523 2013-05-24 22:00:37,402 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,39146,1369432814418-0] handler.CloseRegionHandler(125): Processing close of test,,1369432825000.45883ac87958f956040ff389c9f7eb86. 2013-05-24 22:00:37,403 INFO [RegionServer:1;kiyo.gq1.ygridcore.net,39146,1369432814418] client.HConnectionManager$HConnectionImplementation(1640): Closing zookeeper sessionid=0x13ed88e2cdf0013 2013-05-24 22:00:37,403 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,39146,1369432814418-0] regionserver.HRegion(927): Closing test,,1369432825000.45883ac87958f956040ff389c9f7eb86.: disabling compactions & flushes 2013-05-24 22:00:37,403 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,39146,1369432814418-0] regionserver.HRegion(949): Updates disabled for region test,,1369432825000.45883ac87958f956040ff389c9f7eb86. 2013-05-24 22:00:37,404 INFO [StoreCloserThread-test,,1369432825000.45883ac87958f956040ff389c9f7eb86.-1] regionserver.HStore(663): Closed f 2013-05-24 22:00:37,404 INFO [StoreCloserThread-test,,1369432825000.45883ac87958f956040ff389c9f7eb86.-1] regionserver.HStore(663): Closed norep 2013-05-24 22:00:37,404 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,39146,1369432814418-0] regionserver.HRegion(1006): Closed test,,1369432825000.45883ac87958f956040ff389c9f7eb86. 2013-05-24 22:00:37,404 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,39146,1369432814418-0] handler.CloseRegionHandler(177): Closed region test,,1369432825000.45883ac87958f956040ff389c9f7eb86. 2013-05-24 22:00:37,431 INFO [RegionServer:1;kiyo.gq1.ygridcore.net,39146,1369432814418.periodicFlusher] hbase.Chore(93): RegionServer:1;kiyo.gq1.ygridcore.net,39146,1369432814418.periodicFlusher exiting 2013-05-24 22:00:37,431 INFO [RegionServer:1;kiyo.gq1.ygridcore.net,39146,1369432814418.leaseChecker] regionserver.Leases(124): RegionServer:1;kiyo.gq1.ygridcore.net,39146,1369432814418.leaseChecker closing leases 2013-05-24 22:00:37,431 INFO [RegionServer:1;kiyo.gq1.ygridcore.net,39146,1369432814418.leaseChecker] regionserver.Leases(131): RegionServer:1;kiyo.gq1.ygridcore.net,39146,1369432814418.leaseChecker closed leases 2013-05-24 22:00:37,477 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432834215 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:37,482 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:37,485 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(583): Nothing to replicate, sleeping 100 times 7 2013-05-24 22:00:37,582 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/region-in-transition/a88e8c9e73f832adb3635d6996d9e72d 2013-05-24 22:00:37,583 DEBUG [hbase-am-zkevent-worker-pool-2-thread-9] master.AssignmentManager$4(1189): The znode of region test,,1369432820315.a88e8c9e73f832adb3635d6996d9e72d. has been deleted. 2013-05-24 22:00:37,583 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/region-in-transition 2013-05-24 22:00:37,583 INFO [hbase-am-zkevent-worker-pool-2-thread-9] master.RegionStates(264): Region {NAME => 'test,,1369432820315.a88e8c9e73f832adb3635d6996d9e72d.', STARTKEY => '', ENDKEY => 'bbb', ENCODED => a88e8c9e73f832adb3635d6996d9e72d,} transitioned from {test,,1369432820315.a88e8c9e73f832adb3635d6996d9e72d. state=OPEN, ts=1369432837171, server=kiyo.gq1.ygridcore.net,42690,1369432806911} to {test,,1369432820315.a88e8c9e73f832adb3635d6996d9e72d. state=OPEN, ts=1369432837583, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:37,583 INFO [hbase-am-zkevent-worker-pool-2-thread-9] master.RegionStates(299): Onlined region {NAME => 'test,,1369432820315.a88e8c9e73f832adb3635d6996d9e72d.', STARTKEY => '', ENDKEY => 'bbb', ENCODED => a88e8c9e73f832adb3635d6996d9e72d,} on kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:37,583 INFO [hbase-am-zkevent-worker-pool-2-thread-9] master.AssignmentManager$4(1194): The master has opened the region test,,1369432820315.a88e8c9e73f832adb3635d6996d9e72d. that was online on kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:37,583 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.AssignmentManager(2753): Waiting on null to clear regions-in-transition 2013-05-24 22:00:37,583 DEBUG [MASTER_OPEN_REGION-kiyo.gq1.ygridcore.net,49101,1369432806564-2] zookeeper.ZKAssign(434): master:49101-0x13ed88e2cdf0004 Successfully deleted unassigned node for region a88e8c9e73f832adb3635d6996d9e72d in expected state RS_ZK_REGION_OPENED 2013-05-24 22:00:37,586 DEBUG [hbase-am-zkevent-worker-pool-2-thread-4] master.AssignmentManager(752): Handling transition=RS_ZK_REGION_OPENING, server=kiyo.gq1.ygridcore.net,42690,1369432806911, region=eb38c552d2a930aa1ce862202da80908, current state from region state map ={test,jjj,1369432820320.eb38c552d2a930aa1ce862202da80908. state=PENDING_OPEN, ts=1369432835959, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:37,587 INFO [hbase-am-zkevent-worker-pool-2-thread-4] master.RegionStates(264): Region {NAME => 'test,jjj,1369432820320.eb38c552d2a930aa1ce862202da80908.', STARTKEY => 'jjj', ENDKEY => 'kkk', ENCODED => eb38c552d2a930aa1ce862202da80908,} transitioned from {test,jjj,1369432820320.eb38c552d2a930aa1ce862202da80908. state=PENDING_OPEN, ts=1369432835959, server=kiyo.gq1.ygridcore.net,42690,1369432806911} to {test,jjj,1369432820320.eb38c552d2a930aa1ce862202da80908. state=OPENING, ts=1369432837587, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:37,588 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,53632,1369432814258-0] client.ClientScanner(196): Finished scanning region {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,} 2013-05-24 22:00:37,591 INFO [PostOpenDeployTasks:eb38c552d2a930aa1ce862202da80908] regionserver.HRegionServer(1657): Post open deploy tasks for region=test,jjj,1369432820320.eb38c552d2a930aa1ce862202da80908. 2013-05-24 22:00:37,684 INFO [RegionServer:1;kiyo.gq1.ygridcore.net,39146,1369432814418] snapshot.RegionServerSnapshotManager(151): Stopping RegionServerSnapshotManager abruptly. 2013-05-24 22:00:37,685 INFO [RegionServer:1;kiyo.gq1.ygridcore.net,39146,1369432814418] regionserver.HRegionServer(912): stopping server kiyo.gq1.ygridcore.net,39146,1369432814418; all regions closed. 2013-05-24 22:00:37,685 INFO [RegionServer:1;kiyo.gq1.ygridcore.net,39146,1369432814418.logSyncer] wal.FSHLog$LogSyncer(967): RegionServer:1;kiyo.gq1.ygridcore.net,39146,1369432814418.logSyncer exiting 2013-05-24 22:00:37,685 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,39146,1369432814418] wal.FSHLog(809): closing hlog writer in hdfs://localhost:54104/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,39146,1369432814418 2013-05-24 22:00:37,690 INFO [IPC Server handler 8 on 54104] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33054 is added to blk_1322746500432156304_1014{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:52910|RBW], ReplicaUnderConstruction[127.0.0.1:33054|RBW]]} size 37779 2013-05-24 22:00:37,691 INFO [IPC Server handler 6 on 54104] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52910 is added to blk_1322746500432156304_1014 size 37779 2013-05-24 22:00:37,713 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/region-in-transition/1ce9aafaaa422cef4ce0e1895111c733 2013-05-24 22:00:37,713 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] zookeeper.ZKAssign(862): regionserver:42690-0x13ed88e2cdf0006 Successfully transitioned node 1ce9aafaaa422cef4ce0e1895111c733 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-05-24 22:00:37,713 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/region-in-transition/f852f52d87b0b7acd213f5b49074dde6 2013-05-24 22:00:37,714 INFO [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegion(4139): HRegion.openHRegion Region name ==test,kkk,1369432820320.1ce9aafaaa422cef4ce0e1895111c733. 2013-05-24 22:00:37,714 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegion(4141): Opening region: {NAME => 'test,kkk,1369432820320.1ce9aafaaa422cef4ce0e1895111c733.', STARTKEY => 'kkk', ENDKEY => 'lll', ENCODED => 1ce9aafaaa422cef4ce0e1895111c733,} 2013-05-24 22:00:37,714 DEBUG [hbase-am-zkevent-worker-pool-2-thread-10] master.AssignmentManager$4(1189): The znode of region test,fff,1369432820319.f852f52d87b0b7acd213f5b49074dde6. has been deleted. 2013-05-24 22:00:37,714 INFO [hbase-am-zkevent-worker-pool-2-thread-10] master.RegionStates(264): Region {NAME => 'test,fff,1369432820319.f852f52d87b0b7acd213f5b49074dde6.', STARTKEY => 'fff', ENDKEY => 'ggg', ENCODED => f852f52d87b0b7acd213f5b49074dde6,} transitioned from {test,fff,1369432820319.f852f52d87b0b7acd213f5b49074dde6. state=OPEN, ts=1369432837363, server=kiyo.gq1.ygridcore.net,42690,1369432806911} to {test,fff,1369432820319.f852f52d87b0b7acd213f5b49074dde6. state=OPEN, ts=1369432837714, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:37,714 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,53632,1369432814258-0] zookeeper.ZKUtil(690): master:53632-0x13ed88e2cdf000f Unable to get data of znode /2/recovering-regions/45883ac87958f956040ff389c9f7eb86 because node does not exist (not an error) 2013-05-24 22:00:37,714 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.MetricsRegionSourceImpl(62): Creating new MetricsRegionSourceImpl for table test 1ce9aafaaa422cef4ce0e1895111c733 2013-05-24 22:00:37,714 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/region-in-transition 2013-05-24 22:00:37,715 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegion(518): Instantiated test,kkk,1369432820320.1ce9aafaaa422cef4ce0e1895111c733. 2013-05-24 22:00:37,714 INFO [hbase-am-zkevent-worker-pool-2-thread-10] master.RegionStates(299): Onlined region {NAME => 'test,fff,1369432820319.f852f52d87b0b7acd213f5b49074dde6.', STARTKEY => 'fff', ENDKEY => 'ggg', ENCODED => f852f52d87b0b7acd213f5b49074dde6,} on kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:37,715 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] zookeeper.ZKAssign(862): regionserver:42690-0x13ed88e2cdf0006 Successfully transitioned node 4b3ff3efde12c5afe1220e81d155ac73 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-05-24 22:00:37,715 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.AssignmentManager(2753): Waiting on null to clear regions-in-transition 2013-05-24 22:00:37,715 INFO [hbase-am-zkevent-worker-pool-2-thread-10] master.AssignmentManager$4(1194): The master has opened the region test,fff,1369432820319.f852f52d87b0b7acd213f5b49074dde6. that was online on kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:37,715 DEBUG [MASTER_OPEN_REGION-kiyo.gq1.ygridcore.net,49101,1369432806564-3] zookeeper.ZKAssign(434): master:49101-0x13ed88e2cdf0004 Successfully deleted unassigned node for region f852f52d87b0b7acd213f5b49074dde6 in expected state RS_ZK_REGION_OPENED 2013-05-24 22:00:37,715 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] handler.OpenRegionHandler(373): region transitioned to opened in zookeeper: {NAME => 'test,ggg,1369432820319.4b3ff3efde12c5afe1220e81d155ac73.', STARTKEY => 'ggg', ENDKEY => 'hhh', ENCODED => 4b3ff3efde12c5afe1220e81d155ac73,}, server: kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:37,716 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] handler.OpenRegionHandler(186): Opened test,ggg,1369432820319.4b3ff3efde12c5afe1220e81d155ac73. on server:kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:37,716 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] zookeeper.ZKAssign(786): regionserver:42690-0x13ed88e2cdf0006 Attempting to transition node ab7b9b9773243cff0ac09f3c379e33dc from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-05-24 22:00:37,717 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/region-in-transition/4b3ff3efde12c5afe1220e81d155ac73 2013-05-24 22:00:37,718 INFO [StoreOpenerThread-test,kkk,1369432820320.1ce9aafaaa422cef4ce0e1895111c733.-1] compactions.CompactionConfiguration(85): Compaction configuration size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 604800000, major jitter 0.500000 2013-05-24 22:00:37,720 INFO [StoreOpenerThread-test,kkk,1369432820320.1ce9aafaaa422cef4ce0e1895111c733.-1] compactions.CompactionConfiguration(85): Compaction configuration size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 604800000, major jitter 0.500000 2013-05-24 22:00:37,723 INFO [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegion(608): Onlined test,kkk,1369432820320.1ce9aafaaa422cef4ce0e1895111c733.; next sequenceid=1 2013-05-24 22:00:37,724 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] zookeeper.ZKAssign(598): regionserver:42690-0x13ed88e2cdf0006 Attempting to retransition the opening state of node 1ce9aafaaa422cef4ce0e1895111c733 2013-05-24 22:00:37,730 DEBUG [hbase-am-zkevent-worker-pool-2-thread-12] master.AssignmentManager(752): Handling transition=RS_ZK_REGION_OPENING, server=kiyo.gq1.ygridcore.net,42690,1369432806911, region=1ce9aafaaa422cef4ce0e1895111c733, current state from region state map ={test,kkk,1369432820320.1ce9aafaaa422cef4ce0e1895111c733. state=PENDING_OPEN, ts=1369432835959, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:37,730 INFO [hbase-am-zkevent-worker-pool-2-thread-12] master.RegionStates(264): Region {NAME => 'test,kkk,1369432820320.1ce9aafaaa422cef4ce0e1895111c733.', STARTKEY => 'kkk', ENDKEY => 'lll', ENCODED => 1ce9aafaaa422cef4ce0e1895111c733,} transitioned from {test,kkk,1369432820320.1ce9aafaaa422cef4ce0e1895111c733. state=PENDING_OPEN, ts=1369432835959, server=kiyo.gq1.ygridcore.net,42690,1369432806911} to {test,kkk,1369432820320.1ce9aafaaa422cef4ce0e1895111c733. state=OPENING, ts=1369432837730, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:37,731 DEBUG [hbase-am-zkevent-worker-pool-2-thread-8] master.AssignmentManager(752): Handling transition=RS_ZK_REGION_OPENED, server=kiyo.gq1.ygridcore.net,42690,1369432806911, region=4b3ff3efde12c5afe1220e81d155ac73, current state from region state map ={test,ggg,1369432820319.4b3ff3efde12c5afe1220e81d155ac73. state=OPENING, ts=1369432837010, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:37,732 INFO [hbase-am-zkevent-worker-pool-2-thread-8] master.RegionStates(264): Region {NAME => 'test,ggg,1369432820319.4b3ff3efde12c5afe1220e81d155ac73.', STARTKEY => 'ggg', ENDKEY => 'hhh', ENCODED => 4b3ff3efde12c5afe1220e81d155ac73,} transitioned from {test,ggg,1369432820319.4b3ff3efde12c5afe1220e81d155ac73. state=OPENING, ts=1369432837010, server=kiyo.gq1.ygridcore.net,42690,1369432806911} to {test,ggg,1369432820319.4b3ff3efde12c5afe1220e81d155ac73. state=OPEN, ts=1369432837732, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:37,732 DEBUG [MASTER_OPEN_REGION-kiyo.gq1.ygridcore.net,49101,1369432806564-4] handler.OpenedRegionHandler(146): Handling OPENED event for test,ggg,1369432820319.4b3ff3efde12c5afe1220e81d155ac73. from kiyo.gq1.ygridcore.net,42690,1369432806911; deleting unassigned node 2013-05-24 22:00:37,732 DEBUG [MASTER_OPEN_REGION-kiyo.gq1.ygridcore.net,49101,1369432806564-4] zookeeper.ZKAssign(405): master:49101-0x13ed88e2cdf0004 Deleting existing unassigned node for 4b3ff3efde12c5afe1220e81d155ac73 that is in expected state RS_ZK_REGION_OPENED 2013-05-24 22:00:37,735 INFO [PostOpenDeployTasks:1ce9aafaaa422cef4ce0e1895111c733] regionserver.HRegionServer(1657): Post open deploy tasks for region=test,kkk,1369432820320.1ce9aafaaa422cef4ce0e1895111c733. 2013-05-24 22:00:37,735 DEBUG [ReplicationExecutor-0] replication.ReplicationQueuesZKImpl(379): Creating kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432832342 with data 37443 2013-05-24 22:00:37,746 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/region-in-transition/ab7b9b9773243cff0ac09f3c379e33dc 2013-05-24 22:00:37,746 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] zookeeper.ZKAssign(862): regionserver:42690-0x13ed88e2cdf0006 Successfully transitioned node ab7b9b9773243cff0ac09f3c379e33dc from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-05-24 22:00:37,747 INFO [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(4139): HRegion.openHRegion Region name ==test,mmm,1369432820320.ab7b9b9773243cff0ac09f3c379e33dc. 2013-05-24 22:00:37,747 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(4141): Opening region: {NAME => 'test,mmm,1369432820320.ab7b9b9773243cff0ac09f3c379e33dc.', STARTKEY => 'mmm', ENDKEY => 'nnn', ENCODED => ab7b9b9773243cff0ac09f3c379e33dc,} 2013-05-24 22:00:37,747 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.MetricsRegionSourceImpl(62): Creating new MetricsRegionSourceImpl for table test ab7b9b9773243cff0ac09f3c379e33dc 2013-05-24 22:00:37,747 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(518): Instantiated test,mmm,1369432820320.ab7b9b9773243cff0ac09f3c379e33dc. 2013-05-24 22:00:37,751 INFO [StoreOpenerThread-test,mmm,1369432820320.ab7b9b9773243cff0ac09f3c379e33dc.-1] compactions.CompactionConfiguration(85): Compaction configuration size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 604800000, major jitter 0.500000 2013-05-24 22:00:37,753 INFO [StoreOpenerThread-test,mmm,1369432820320.ab7b9b9773243cff0ac09f3c379e33dc.-1] compactions.CompactionConfiguration(85): Compaction configuration size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 604800000, major jitter 0.500000 2013-05-24 22:00:37,754 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:42690-0x13ed88e2cdf0006 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/recovering-regions/eb38c552d2a930aa1ce862202da80908 2013-05-24 22:00:37,755 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/region-in-transition/4b3ff3efde12c5afe1220e81d155ac73 2013-05-24 22:00:37,755 DEBUG [hbase-am-zkevent-worker-pool-2-thread-16] master.AssignmentManager$4(1189): The znode of region test,ggg,1369432820319.4b3ff3efde12c5afe1220e81d155ac73. has been deleted. 2013-05-24 22:00:37,755 INFO [hbase-am-zkevent-worker-pool-2-thread-16] master.RegionStates(264): Region {NAME => 'test,ggg,1369432820319.4b3ff3efde12c5afe1220e81d155ac73.', STARTKEY => 'ggg', ENDKEY => 'hhh', ENCODED => 4b3ff3efde12c5afe1220e81d155ac73,} transitioned from {test,ggg,1369432820319.4b3ff3efde12c5afe1220e81d155ac73. state=OPEN, ts=1369432837732, server=kiyo.gq1.ygridcore.net,42690,1369432806911} to {test,ggg,1369432820319.4b3ff3efde12c5afe1220e81d155ac73. state=OPEN, ts=1369432837755, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:37,755 INFO [hbase-am-zkevent-worker-pool-2-thread-16] master.RegionStates(299): Onlined region {NAME => 'test,ggg,1369432820319.4b3ff3efde12c5afe1220e81d155ac73.', STARTKEY => 'ggg', ENDKEY => 'hhh', ENCODED => 4b3ff3efde12c5afe1220e81d155ac73,} on kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:37,755 INFO [hbase-am-zkevent-worker-pool-2-thread-16] master.AssignmentManager$4(1194): The master has opened the region test,ggg,1369432820319.4b3ff3efde12c5afe1220e81d155ac73. that was online on kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:37,755 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.AssignmentManager(2753): Waiting on null to clear regions-in-transition 2013-05-24 22:00:37,756 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/region-in-transition 2013-05-24 22:00:37,757 DEBUG [MASTER_OPEN_REGION-kiyo.gq1.ygridcore.net,49101,1369432806564-4] zookeeper.ZKAssign(434): master:49101-0x13ed88e2cdf0004 Successfully deleted unassigned node for region 4b3ff3efde12c5afe1220e81d155ac73 in expected state RS_ZK_REGION_OPENED 2013-05-24 22:00:37,757 INFO [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(608): Onlined test,mmm,1369432820320.ab7b9b9773243cff0ac09f3c379e33dc.; next sequenceid=1 2013-05-24 22:00:37,757 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] zookeeper.ZKAssign(598): regionserver:42690-0x13ed88e2cdf0006 Attempting to retransition the opening state of node ab7b9b9773243cff0ac09f3c379e33dc 2013-05-24 22:00:37,758 DEBUG [hbase-am-zkevent-worker-pool-2-thread-13] master.AssignmentManager(752): Handling transition=RS_ZK_REGION_OPENING, server=kiyo.gq1.ygridcore.net,42690,1369432806911, region=ab7b9b9773243cff0ac09f3c379e33dc, current state from region state map ={test,mmm,1369432820320.ab7b9b9773243cff0ac09f3c379e33dc. state=PENDING_OPEN, ts=1369432835959, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:37,758 INFO [hbase-am-zkevent-worker-pool-2-thread-13] master.RegionStates(264): Region {NAME => 'test,mmm,1369432820320.ab7b9b9773243cff0ac09f3c379e33dc.', STARTKEY => 'mmm', ENDKEY => 'nnn', ENCODED => ab7b9b9773243cff0ac09f3c379e33dc,} transitioned from {test,mmm,1369432820320.ab7b9b9773243cff0ac09f3c379e33dc. state=PENDING_OPEN, ts=1369432835959, server=kiyo.gq1.ygridcore.net,42690,1369432806911} to {test,mmm,1369432820320.ab7b9b9773243cff0ac09f3c379e33dc. state=OPENING, ts=1369432837758, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:37,763 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,53632,1369432814258-0] master.SplitLogManager(1117): Mark region 45883ac87958f956040ff389c9f7eb86 recovering from failed region server kiyo.gq1.ygridcore.net,39146,1369432814418 2013-05-24 22:00:37,763 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,53632,1369432814258-0] handler.ServerShutdownHandler(205): Reassigning 1 region(s) that kiyo.gq1.ygridcore.net,39146,1369432814418 was carrying (and 0 regions(s) that were opening on this server) 2013-05-24 22:00:37,763 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,53632,1369432814258-0] master.AssignmentManager(2411): Not use bulk assigning since we are assigning only 1 region(s) to 1 server(s) 2013-05-24 22:00:37,763 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,53632,1369432814258-0] master.AssignmentManager(1474): Bulk assigning 1 region(s) to kiyo.gq1.ygridcore.net,51523,1369432814392 2013-05-24 22:00:37,764 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,53632,1369432814258-0] master.RegionStates(325): Offline a region with current state=OPEN, expected state=OFFLINE, assigned to server: kiyo.gq1.ygridcore.net,39146,1369432814418, expected null 2013-05-24 22:00:37,764 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,53632,1369432814258-0] master.RegionStates(264): Region {NAME => 'test,,1369432825000.45883ac87958f956040ff389c9f7eb86.', STARTKEY => '', ENDKEY => '', ENCODED => 45883ac87958f956040ff389c9f7eb86,} transitioned from {test,,1369432825000.45883ac87958f956040ff389c9f7eb86. state=OPEN, ts=1369432826109, server=kiyo.gq1.ygridcore.net,39146,1369432814418} to {test,,1369432825000.45883ac87958f956040ff389c9f7eb86. state=OFFLINE, ts=1369432837764, server=null} 2013-05-24 22:00:37,764 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,53632,1369432814258-0] master.RegionStates(264): Region {NAME => 'test,,1369432825000.45883ac87958f956040ff389c9f7eb86.', STARTKEY => '', ENDKEY => '', ENCODED => 45883ac87958f956040ff389c9f7eb86,} transitioned from {test,,1369432825000.45883ac87958f956040ff389c9f7eb86. state=OFFLINE, ts=1369432837764, server=null} to {test,,1369432825000.45883ac87958f956040ff389c9f7eb86. state=OFFLINE, ts=1369432837764, server=null} 2013-05-24 22:00:37,764 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,53632,1369432814258-0] zookeeper.ZKAssign(177): master:53632-0x13ed88e2cdf000f Async create of unassigned node for 45883ac87958f956040ff389c9f7eb86 with OFFLINE state 2013-05-24 22:00:37,771 DEBUG [PostOpenDeployTasks:eb38c552d2a930aa1ce862202da80908] regionserver.HRegionServer(4268): Update last flushed sequence id of region eb38c552d2a930aa1ce862202da80908 for kiyo.gq1.ygridcore.net,49701,1369432806997 2013-05-24 22:00:37,772 DEBUG [ReplicationExecutor-0] replication.ReplicationQueuesZKImpl(379): Creating kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432833549 with data 0 2013-05-24 22:00:37,774 INFO [PostOpenDeployTasks:ab7b9b9773243cff0ac09f3c379e33dc] regionserver.HRegionServer(1657): Post open deploy tasks for region=test,mmm,1369432820320.ab7b9b9773243cff0ac09f3c379e33dc. 2013-05-24 22:00:37,790 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:53632-0x13ed88e2cdf000f Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/2/region-in-transition 2013-05-24 22:00:37,790 DEBUG [pool-1-thread-1-EventThread] master.OfflineCallback(69): rs={test,,1369432825000.45883ac87958f956040ff389c9f7eb86. state=OFFLINE, ts=1369432837764, server=null}, server=kiyo.gq1.ygridcore.net,51523,1369432814392 2013-05-24 22:00:37,796 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:42690-0x13ed88e2cdf0006 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/recovering-regions/1ce9aafaaa422cef4ce0e1895111c733 2013-05-24 22:00:37,797 DEBUG [pool-1-thread-1-EventThread] master.OfflineCallback$ExistCallback(106): rs={test,,1369432825000.45883ac87958f956040ff389c9f7eb86. state=OFFLINE, ts=1369432837764, server=null}, server=kiyo.gq1.ygridcore.net,51523,1369432814392 2013-05-24 22:00:37,799 INFO [PostOpenDeployTasks:eb38c552d2a930aa1ce862202da80908] catalog.MetaEditor(432): Updated row test,jjj,1369432820320.eb38c552d2a930aa1ce862202da80908. with server=kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:37,799 INFO [PostOpenDeployTasks:eb38c552d2a930aa1ce862202da80908] regionserver.HRegionServer(1682): Done with post open deploy task for region=test,jjj,1369432820320.eb38c552d2a930aa1ce862202da80908. 2013-05-24 22:00:37,800 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] zookeeper.ZKAssign(786): regionserver:42690-0x13ed88e2cdf0006 Attempting to transition node eb38c552d2a930aa1ce862202da80908 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-05-24 22:00:37,800 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,53632,1369432814258-0] master.AssignmentManager(1512): kiyo.gq1.ygridcore.net,51523,1369432814392 unassigned znodes=1 of total=1 2013-05-24 22:00:37,800 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,53632,1369432814258-0] master.RegionStates(264): Region {NAME => 'test,,1369432825000.45883ac87958f956040ff389c9f7eb86.', STARTKEY => '', ENDKEY => '', ENCODED => 45883ac87958f956040ff389c9f7eb86,} transitioned from {test,,1369432825000.45883ac87958f956040ff389c9f7eb86. state=OFFLINE, ts=1369432837764, server=null} to {test,,1369432825000.45883ac87958f956040ff389c9f7eb86. state=PENDING_OPEN, ts=1369432837800, server=kiyo.gq1.ygridcore.net,51523,1369432814392} 2013-05-24 22:00:37,802 INFO [PRI IPC Server handler 5 on 51523] regionserver.HRegionServer(3454): Received request to open region: test,,1369432825000.45883ac87958f956040ff389c9f7eb86. on kiyo.gq1.ygridcore.net,51523,1369432814392 2013-05-24 22:00:37,805 DEBUG [PostOpenDeployTasks:1ce9aafaaa422cef4ce0e1895111c733] regionserver.HRegionServer(4268): Update last flushed sequence id of region 1ce9aafaaa422cef4ce0e1895111c733 for kiyo.gq1.ygridcore.net,49701,1369432806997 2013-05-24 22:00:37,807 DEBUG [ReplicationExecutor-0] replication.ReplicationQueuesZKImpl(379): Creating kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432829856 with data 46226 2013-05-24 22:00:37,809 INFO [PostOpenDeployTasks:1ce9aafaaa422cef4ce0e1895111c733] catalog.MetaEditor(432): Updated row test,kkk,1369432820320.1ce9aafaaa422cef4ce0e1895111c733. with server=kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:37,809 INFO [PostOpenDeployTasks:1ce9aafaaa422cef4ce0e1895111c733] regionserver.HRegionServer(1682): Done with post open deploy task for region=test,kkk,1369432820320.1ce9aafaaa422cef4ce0e1895111c733. 2013-05-24 22:00:37,809 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] zookeeper.ZKAssign(786): regionserver:42690-0x13ed88e2cdf0006 Attempting to transition node 1ce9aafaaa422cef4ce0e1895111c733 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-05-24 22:00:37,819 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/region-in-transition/eb38c552d2a930aa1ce862202da80908 2013-05-24 22:00:37,819 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,51523,1369432814392-0] zookeeper.ZKAssign(786): regionserver:51523-0x13ed88e2cdf0010 Attempting to transition node 45883ac87958f956040ff389c9f7eb86 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-05-24 22:00:37,819 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,53632,1369432814258-0] master.AssignmentManager(1634): Bulk assigning done for kiyo.gq1.ygridcore.net,51523,1369432814392 2013-05-24 22:00:37,819 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] zookeeper.ZKAssign(862): regionserver:42690-0x13ed88e2cdf0006 Successfully transitioned node eb38c552d2a930aa1ce862202da80908 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-05-24 22:00:37,819 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,53632,1369432814258-0] master.AssignmentManager(2753): Waiting on null to clear regions-in-transition 2013-05-24 22:00:37,820 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] handler.OpenRegionHandler(373): region transitioned to opened in zookeeper: {NAME => 'test,jjj,1369432820320.eb38c552d2a930aa1ce862202da80908.', STARTKEY => 'jjj', ENDKEY => 'kkk', ENCODED => eb38c552d2a930aa1ce862202da80908,}, server: kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:37,820 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] handler.OpenRegionHandler(186): Opened test,jjj,1369432820320.eb38c552d2a930aa1ce862202da80908. on server:kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:37,820 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] zookeeper.ZKAssign(786): regionserver:42690-0x13ed88e2cdf0006 Attempting to transition node 53e3aef4cc8815dd31923659bff5ca21 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-05-24 22:00:37,830 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:42690-0x13ed88e2cdf0006 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/recovering-regions/ab7b9b9773243cff0ac09f3c379e33dc 2013-05-24 22:00:37,831 DEBUG [hbase-am-zkevent-worker-pool-2-thread-17] master.AssignmentManager(752): Handling transition=RS_ZK_REGION_OPENED, server=kiyo.gq1.ygridcore.net,42690,1369432806911, region=eb38c552d2a930aa1ce862202da80908, current state from region state map ={test,jjj,1369432820320.eb38c552d2a930aa1ce862202da80908. state=OPENING, ts=1369432837587, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:37,831 INFO [hbase-am-zkevent-worker-pool-2-thread-17] master.RegionStates(264): Region {NAME => 'test,jjj,1369432820320.eb38c552d2a930aa1ce862202da80908.', STARTKEY => 'jjj', ENDKEY => 'kkk', ENCODED => eb38c552d2a930aa1ce862202da80908,} transitioned from {test,jjj,1369432820320.eb38c552d2a930aa1ce862202da80908. state=OPENING, ts=1369432837587, server=kiyo.gq1.ygridcore.net,42690,1369432806911} to {test,jjj,1369432820320.eb38c552d2a930aa1ce862202da80908. state=OPEN, ts=1369432837831, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:37,831 DEBUG [MASTER_OPEN_REGION-kiyo.gq1.ygridcore.net,49101,1369432806564-0] handler.OpenedRegionHandler(146): Handling OPENED event for test,jjj,1369432820320.eb38c552d2a930aa1ce862202da80908. from kiyo.gq1.ygridcore.net,42690,1369432806911; deleting unassigned node 2013-05-24 22:00:37,831 DEBUG [MASTER_OPEN_REGION-kiyo.gq1.ygridcore.net,49101,1369432806564-0] zookeeper.ZKAssign(405): master:49101-0x13ed88e2cdf0004 Deleting existing unassigned node for eb38c552d2a930aa1ce862202da80908 that is in expected state RS_ZK_REGION_OPENED 2013-05-24 22:00:37,852 DEBUG [PostOpenDeployTasks:ab7b9b9773243cff0ac09f3c379e33dc] regionserver.HRegionServer(4268): Update last flushed sequence id of region ab7b9b9773243cff0ac09f3c379e33dc for kiyo.gq1.ygridcore.net,49701,1369432806997 2013-05-24 22:00:37,920 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:53632-0x13ed88e2cdf000f Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/2/region-in-transition/45883ac87958f956040ff389c9f7eb86 2013-05-24 22:00:37,920 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/region-in-transition/1ce9aafaaa422cef4ce0e1895111c733 2013-05-24 22:00:37,921 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] zookeeper.ZKAssign(862): regionserver:42690-0x13ed88e2cdf0006 Successfully transitioned node 1ce9aafaaa422cef4ce0e1895111c733 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-05-24 22:00:37,921 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] handler.OpenRegionHandler(373): region transitioned to opened in zookeeper: {NAME => 'test,kkk,1369432820320.1ce9aafaaa422cef4ce0e1895111c733.', STARTKEY => 'kkk', ENDKEY => 'lll', ENCODED => 1ce9aafaaa422cef4ce0e1895111c733,}, server: kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:37,921 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] handler.OpenRegionHandler(186): Opened test,kkk,1369432820320.1ce9aafaaa422cef4ce0e1895111c733. on server:kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:37,921 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] zookeeper.ZKAssign(786): regionserver:42690-0x13ed88e2cdf0006 Attempting to transition node 0d6df81a73ad3b2a79add80716da62ac from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-05-24 22:00:37,921 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] zookeeper.ZKAssign(862): regionserver:42690-0x13ed88e2cdf0006 Successfully transitioned node 53e3aef4cc8815dd31923659bff5ca21 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-05-24 22:00:37,921 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/region-in-transition/53e3aef4cc8815dd31923659bff5ca21 2013-05-24 22:00:37,922 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,51523,1369432814392-0] zookeeper.ZKAssign(862): regionserver:51523-0x13ed88e2cdf0010 Successfully transitioned node 45883ac87958f956040ff389c9f7eb86 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-05-24 22:00:37,922 INFO [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegion(4139): HRegion.openHRegion Region name ==test,nnn,1369432820320.53e3aef4cc8815dd31923659bff5ca21. 2013-05-24 22:00:37,922 INFO [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,51523,1369432814392-0] regionserver.HRegion(4139): HRegion.openHRegion Region name ==test,,1369432825000.45883ac87958f956040ff389c9f7eb86. 2013-05-24 22:00:37,922 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegion(4141): Opening region: {NAME => 'test,nnn,1369432820320.53e3aef4cc8815dd31923659bff5ca21.', STARTKEY => 'nnn', ENDKEY => 'ooo', ENCODED => 53e3aef4cc8815dd31923659bff5ca21,} 2013-05-24 22:00:37,922 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,51523,1369432814392-0] regionserver.HRegion(4141): Opening region: {NAME => 'test,,1369432825000.45883ac87958f956040ff389c9f7eb86.', STARTKEY => '', ENDKEY => '', ENCODED => 45883ac87958f956040ff389c9f7eb86,} 2013-05-24 22:00:37,922 DEBUG [ReplicationExecutor-0] replication.ReplicationQueuesZKImpl(379): Creating kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432809419 with data 68784 2013-05-24 22:00:37,922 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,51523,1369432814392-0] regionserver.MetricsRegionSourceImpl(62): Creating new MetricsRegionSourceImpl for table test 45883ac87958f956040ff389c9f7eb86 2013-05-24 22:00:37,922 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.MetricsRegionSourceImpl(62): Creating new MetricsRegionSourceImpl for table test 53e3aef4cc8815dd31923659bff5ca21 2013-05-24 22:00:37,923 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,51523,1369432814392-0] regionserver.HRegion(518): Instantiated test,,1369432825000.45883ac87958f956040ff389c9f7eb86. 2013-05-24 22:00:37,923 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegion(518): Instantiated test,nnn,1369432820320.53e3aef4cc8815dd31923659bff5ca21. 2013-05-24 22:00:37,926 INFO [StoreOpenerThread-test,nnn,1369432820320.53e3aef4cc8815dd31923659bff5ca21.-1] compactions.CompactionConfiguration(85): Compaction configuration size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 604800000, major jitter 0.500000 2013-05-24 22:00:37,926 INFO [StoreOpenerThread-test,,1369432825000.45883ac87958f956040ff389c9f7eb86.-1] compactions.CompactionConfiguration(85): Compaction configuration size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 604800000, major jitter 0.500000 2013-05-24 22:00:37,929 INFO [StoreOpenerThread-test,nnn,1369432820320.53e3aef4cc8815dd31923659bff5ca21.-1] compactions.CompactionConfiguration(85): Compaction configuration size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 604800000, major jitter 0.500000 2013-05-24 22:00:37,929 INFO [StoreOpenerThread-test,,1369432825000.45883ac87958f956040ff389c9f7eb86.-1] compactions.CompactionConfiguration(85): Compaction configuration size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 604800000, major jitter 0.500000 2013-05-24 22:00:37,933 INFO [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegion(608): Onlined test,nnn,1369432820320.53e3aef4cc8815dd31923659bff5ca21.; next sequenceid=1 2013-05-24 22:00:37,933 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] zookeeper.ZKAssign(598): regionserver:42690-0x13ed88e2cdf0006 Attempting to retransition the opening state of node 53e3aef4cc8815dd31923659bff5ca21 2013-05-24 22:00:37,933 INFO [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,51523,1369432814392-0] regionserver.HRegion(608): Onlined test,,1369432825000.45883ac87958f956040ff389c9f7eb86.; next sequenceid=1 2013-05-24 22:00:37,933 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,51523,1369432814392-0] zookeeper.ZKAssign(598): regionserver:51523-0x13ed88e2cdf0010 Attempting to retransition the opening state of node 45883ac87958f956040ff389c9f7eb86 2013-05-24 22:00:38,021 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/region-in-transition/eb38c552d2a930aa1ce862202da80908 2013-05-24 22:00:38,021 DEBUG [hbase-am-zkevent-worker-pool-2-thread-14] master.AssignmentManager$4(1189): The znode of region test,jjj,1369432820320.eb38c552d2a930aa1ce862202da80908. has been deleted. 2013-05-24 22:00:38,021 INFO [hbase-am-zkevent-worker-pool-2-thread-14] master.RegionStates(264): Region {NAME => 'test,jjj,1369432820320.eb38c552d2a930aa1ce862202da80908.', STARTKEY => 'jjj', ENDKEY => 'kkk', ENCODED => eb38c552d2a930aa1ce862202da80908,} transitioned from {test,jjj,1369432820320.eb38c552d2a930aa1ce862202da80908. state=OPEN, ts=1369432837831, server=kiyo.gq1.ygridcore.net,42690,1369432806911} to {test,jjj,1369432820320.eb38c552d2a930aa1ce862202da80908. state=OPEN, ts=1369432838021, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:38,021 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/region-in-transition 2013-05-24 22:00:38,021 INFO [hbase-am-zkevent-worker-pool-2-thread-14] master.RegionStates(299): Onlined region {NAME => 'test,jjj,1369432820320.eb38c552d2a930aa1ce862202da80908.', STARTKEY => 'jjj', ENDKEY => 'kkk', ENCODED => eb38c552d2a930aa1ce862202da80908,} on kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:38,021 INFO [hbase-am-zkevent-worker-pool-2-thread-14] master.AssignmentManager$4(1194): The master has opened the region test,jjj,1369432820320.eb38c552d2a930aa1ce862202da80908. that was online on kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:38,022 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.AssignmentManager(2753): Waiting on null to clear regions-in-transition 2013-05-24 22:00:38,022 DEBUG [hbase-am-zkevent-worker-pool-13-thread-10] master.AssignmentManager(752): Handling transition=RS_ZK_REGION_OPENING, server=kiyo.gq1.ygridcore.net,51523,1369432814392, region=45883ac87958f956040ff389c9f7eb86, current state from region state map ={test,,1369432825000.45883ac87958f956040ff389c9f7eb86. state=PENDING_OPEN, ts=1369432837800, server=kiyo.gq1.ygridcore.net,51523,1369432814392} 2013-05-24 22:00:38,022 INFO [hbase-am-zkevent-worker-pool-13-thread-10] master.RegionStates(264): Region {NAME => 'test,,1369432825000.45883ac87958f956040ff389c9f7eb86.', STARTKEY => '', ENDKEY => '', ENCODED => 45883ac87958f956040ff389c9f7eb86,} transitioned from {test,,1369432825000.45883ac87958f956040ff389c9f7eb86. state=PENDING_OPEN, ts=1369432837800, server=kiyo.gq1.ygridcore.net,51523,1369432814392} to {test,,1369432825000.45883ac87958f956040ff389c9f7eb86. state=OPENING, ts=1369432838022, server=kiyo.gq1.ygridcore.net,51523,1369432814392} 2013-05-24 22:00:38,022 DEBUG [MASTER_OPEN_REGION-kiyo.gq1.ygridcore.net,49101,1369432806564-0] zookeeper.ZKAssign(434): master:49101-0x13ed88e2cdf0004 Successfully deleted unassigned node for region eb38c552d2a930aa1ce862202da80908 in expected state RS_ZK_REGION_OPENED 2013-05-24 22:00:38,025 DEBUG [hbase-am-zkevent-worker-pool-2-thread-18] master.AssignmentManager(752): Handling transition=RS_ZK_REGION_OPENED, server=kiyo.gq1.ygridcore.net,42690,1369432806911, region=1ce9aafaaa422cef4ce0e1895111c733, current state from region state map ={test,kkk,1369432820320.1ce9aafaaa422cef4ce0e1895111c733. state=OPENING, ts=1369432837730, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:38,025 INFO [PostOpenDeployTasks:ab7b9b9773243cff0ac09f3c379e33dc] catalog.MetaEditor(432): Updated row test,mmm,1369432820320.ab7b9b9773243cff0ac09f3c379e33dc. with server=kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:38,026 INFO [PostOpenDeployTasks:ab7b9b9773243cff0ac09f3c379e33dc] regionserver.HRegionServer(1682): Done with post open deploy task for region=test,mmm,1369432820320.ab7b9b9773243cff0ac09f3c379e33dc. 2013-05-24 22:00:38,025 INFO [hbase-am-zkevent-worker-pool-2-thread-18] master.RegionStates(264): Region {NAME => 'test,kkk,1369432820320.1ce9aafaaa422cef4ce0e1895111c733.', STARTKEY => 'kkk', ENDKEY => 'lll', ENCODED => 1ce9aafaaa422cef4ce0e1895111c733,} transitioned from {test,kkk,1369432820320.1ce9aafaaa422cef4ce0e1895111c733. state=OPENING, ts=1369432837730, server=kiyo.gq1.ygridcore.net,42690,1369432806911} to {test,kkk,1369432820320.1ce9aafaaa422cef4ce0e1895111c733. state=OPEN, ts=1369432838025, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:38,026 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] zookeeper.ZKAssign(786): regionserver:42690-0x13ed88e2cdf0006 Attempting to transition node ab7b9b9773243cff0ac09f3c379e33dc from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-05-24 22:00:38,026 DEBUG [hbase-am-zkevent-worker-pool-2-thread-1] master.AssignmentManager(752): Handling transition=RS_ZK_REGION_OPENING, server=kiyo.gq1.ygridcore.net,42690,1369432806911, region=53e3aef4cc8815dd31923659bff5ca21, current state from region state map ={test,nnn,1369432820320.53e3aef4cc8815dd31923659bff5ca21. state=PENDING_OPEN, ts=1369432835959, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:38,026 DEBUG [MASTER_OPEN_REGION-kiyo.gq1.ygridcore.net,49101,1369432806564-1] handler.OpenedRegionHandler(146): Handling OPENED event for test,kkk,1369432820320.1ce9aafaaa422cef4ce0e1895111c733. from kiyo.gq1.ygridcore.net,42690,1369432806911; deleting unassigned node 2013-05-24 22:00:38,026 INFO [hbase-am-zkevent-worker-pool-2-thread-1] master.RegionStates(264): Region {NAME => 'test,nnn,1369432820320.53e3aef4cc8815dd31923659bff5ca21.', STARTKEY => 'nnn', ENDKEY => 'ooo', ENCODED => 53e3aef4cc8815dd31923659bff5ca21,} transitioned from {test,nnn,1369432820320.53e3aef4cc8815dd31923659bff5ca21. state=PENDING_OPEN, ts=1369432835959, server=kiyo.gq1.ygridcore.net,42690,1369432806911} to {test,nnn,1369432820320.53e3aef4cc8815dd31923659bff5ca21. state=OPENING, ts=1369432838026, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:38,026 DEBUG [MASTER_OPEN_REGION-kiyo.gq1.ygridcore.net,49101,1369432806564-1] zookeeper.ZKAssign(405): master:49101-0x13ed88e2cdf0004 Deleting existing unassigned node for 1ce9aafaaa422cef4ce0e1895111c733 that is in expected state RS_ZK_REGION_OPENED 2013-05-24 22:00:38,188 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432834215 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:38,192 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:38,195 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(583): Nothing to replicate, sleeping 100 times 8 2013-05-24 22:00:38,227 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,39146,1369432814418-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:39146-0x13ed88e2cdf0011 Received ZooKeeper Event, type=None, state=Expired, path=null 2013-05-24 22:00:38,228 FATAL [RegionServer:1;kiyo.gq1.ygridcore.net,39146,1369432814418-EventThread] regionserver.HRegionServer(1705): ABORTING region server kiyo.gq1.ygridcore.net,39146,1369432814418: regionserver:39146-0x13ed88e2cdf0011 regionserver:39146-0x13ed88e2cdf0011 received expired from ZooKeeper, aborting org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:398) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:316) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495) 2013-05-24 22:00:38,228 FATAL [RegionServer:1;kiyo.gq1.ygridcore.net,39146,1369432814418-EventThread] regionserver.HRegionServer(1713): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2013-05-24 22:00:38,230 ERROR [IPC Server handler 2 on 53632] master.HMaster(1246): Region server kiyo.gq1.ygridcore.net,39146,1369432814418 reported a fatal error: ABORTING region server kiyo.gq1.ygridcore.net,39146,1369432814418: regionserver:39146-0x13ed88e2cdf0011 regionserver:39146-0x13ed88e2cdf0011 received expired from ZooKeeper, aborting Cause: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:398) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:316) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495) 2013-05-24 22:00:38,231 INFO [RegionServer:1;kiyo.gq1.ygridcore.net,39146,1369432814418-EventThread] regionserver.HRegionServer(1639): STOPPED: regionserver:39146-0x13ed88e2cdf0011 regionserver:39146-0x13ed88e2cdf0011 received expired from ZooKeeper, aborting 2013-05-24 22:00:38,274 INFO [PostOpenDeployTasks:45883ac87958f956040ff389c9f7eb86] regionserver.HRegionServer(1657): Post open deploy tasks for region=test,,1369432825000.45883ac87958f956040ff389c9f7eb86. 2013-05-24 22:00:38,275 INFO [PostOpenDeployTasks:53e3aef4cc8815dd31923659bff5ca21] regionserver.HRegionServer(1657): Post open deploy tasks for region=test,nnn,1369432820320.53e3aef4cc8815dd31923659bff5ca21. 2013-05-24 22:00:38,342 INFO [kiyo.gq1.ygridcore.net,39146,1369432814418Replication Statistics #0] regionserver.Replication$ReplicationStatisticsThread(291): Sink: age in ms of last applied edit: 3247, total replicated edits: 2115 2013-05-24 22:00:38,353 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/region-in-transition/0d6df81a73ad3b2a79add80716da62ac 2013-05-24 22:00:38,354 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] zookeeper.ZKAssign(862): regionserver:42690-0x13ed88e2cdf0006 Successfully transitioned node 0d6df81a73ad3b2a79add80716da62ac from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-05-24 22:00:38,354 INFO [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegion(4139): HRegion.openHRegion Region name ==test,ooo,1369432820320.0d6df81a73ad3b2a79add80716da62ac. 2013-05-24 22:00:38,354 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegion(4141): Opening region: {NAME => 'test,ooo,1369432820320.0d6df81a73ad3b2a79add80716da62ac.', STARTKEY => 'ooo', ENDKEY => 'ppp', ENCODED => 0d6df81a73ad3b2a79add80716da62ac,} 2013-05-24 22:00:38,355 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.MetricsRegionSourceImpl(62): Creating new MetricsRegionSourceImpl for table test 0d6df81a73ad3b2a79add80716da62ac 2013-05-24 22:00:38,355 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegion(518): Instantiated test,ooo,1369432820320.0d6df81a73ad3b2a79add80716da62ac. 2013-05-24 22:00:38,356 DEBUG [hbase-am-zkevent-worker-pool-2-thread-20] master.AssignmentManager(752): Handling transition=RS_ZK_REGION_OPENING, server=kiyo.gq1.ygridcore.net,42690,1369432806911, region=0d6df81a73ad3b2a79add80716da62ac, current state from region state map ={test,ooo,1369432820320.0d6df81a73ad3b2a79add80716da62ac. state=PENDING_OPEN, ts=1369432835959, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:38,356 INFO [hbase-am-zkevent-worker-pool-2-thread-20] master.RegionStates(264): Region {NAME => 'test,ooo,1369432820320.0d6df81a73ad3b2a79add80716da62ac.', STARTKEY => 'ooo', ENDKEY => 'ppp', ENCODED => 0d6df81a73ad3b2a79add80716da62ac,} transitioned from {test,ooo,1369432820320.0d6df81a73ad3b2a79add80716da62ac. state=PENDING_OPEN, ts=1369432835959, server=kiyo.gq1.ygridcore.net,42690,1369432806911} to {test,ooo,1369432820320.0d6df81a73ad3b2a79add80716da62ac. state=OPENING, ts=1369432838356, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:38,359 INFO [StoreOpenerThread-test,ooo,1369432820320.0d6df81a73ad3b2a79add80716da62ac.-1] compactions.CompactionConfiguration(85): Compaction configuration size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 604800000, major jitter 0.500000 2013-05-24 22:00:38,361 INFO [StoreOpenerThread-test,ooo,1369432820320.0d6df81a73ad3b2a79add80716da62ac.-1] compactions.CompactionConfiguration(85): Compaction configuration size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 604800000, major jitter 0.500000 2013-05-24 22:00:38,364 INFO [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegion(608): Onlined test,ooo,1369432820320.0d6df81a73ad3b2a79add80716da62ac.; next sequenceid=1 2013-05-24 22:00:38,364 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] zookeeper.ZKAssign(598): regionserver:42690-0x13ed88e2cdf0006 Attempting to retransition the opening state of node 0d6df81a73ad3b2a79add80716da62ac 2013-05-24 22:00:38,399 INFO [kiyo.gq1.ygridcore.net,51523,1369432814392Replication Statistics #0] regionserver.Replication$ReplicationStatisticsThread(291): Sink: age in ms of last applied edit: 4797, total replicated edits: 1629 2013-05-24 22:00:38,453 INFO [RegionServer:1;kiyo.gq1.ygridcore.net,39146,1369432814418] regionserver.Leases(124): RegionServer:1;kiyo.gq1.ygridcore.net,39146,1369432814418 closing leases 2013-05-24 22:00:38,454 INFO [RegionServer:1;kiyo.gq1.ygridcore.net,39146,1369432814418] regionserver.Leases(131): RegionServer:1;kiyo.gq1.ygridcore.net,39146,1369432814418 closed leases 2013-05-24 22:00:38,454 INFO [RegionServer:1;kiyo.gq1.ygridcore.net,39146,1369432814418] regionserver.CompactSplitThread(318): Waiting for Split Thread to finish... 2013-05-24 22:00:38,454 INFO [RegionServer:1;kiyo.gq1.ygridcore.net,39146,1369432814418] regionserver.CompactSplitThread(318): Waiting for Merge Thread to finish... 2013-05-24 22:00:38,454 INFO [RegionServer:1;kiyo.gq1.ygridcore.net,39146,1369432814418] regionserver.CompactSplitThread(318): Waiting for Large Compaction Thread to finish... 2013-05-24 22:00:38,454 INFO [RegionServer:1;kiyo.gq1.ygridcore.net,39146,1369432814418] regionserver.CompactSplitThread(318): Waiting for Small Compaction Thread to finish... 2013-05-24 22:00:38,454 WARN [RegionServer:1;kiyo.gq1.ygridcore.net,39146,1369432814418] zookeeper.RecoverableZooKeeper(237): Possibly transient ZooKeeper, quorum=localhost:51411, exception=org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /2/replication/rs/kiyo.gq1.ygridcore.net,39146,1369432814418 2013-05-24 22:00:38,455 INFO [RegionServer:1;kiyo.gq1.ygridcore.net,39146,1369432814418] util.RetryCounter(54): Sleeping 20ms before retry #1... 2013-05-24 22:00:38,475 WARN [RegionServer:1;kiyo.gq1.ygridcore.net,39146,1369432814418] zookeeper.RecoverableZooKeeper(237): Possibly transient ZooKeeper, quorum=localhost:51411, exception=org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /2/replication/rs/kiyo.gq1.ygridcore.net,39146,1369432814418 2013-05-24 22:00:38,475 ERROR [RegionServer:1;kiyo.gq1.ygridcore.net,39146,1369432814418] zookeeper.RecoverableZooKeeper(239): ZooKeeper getChildren failed after 1 retries 2013-05-24 22:00:38,476 INFO [RegionServer:1;kiyo.gq1.ygridcore.net,39146,1369432814418] client.HConnectionManager$HConnectionImplementation(1640): Closing zookeeper sessionid=0x13ed88e2cdf0016 2013-05-24 22:00:38,495 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/region-in-transition/1ce9aafaaa422cef4ce0e1895111c733 2013-05-24 22:00:38,495 DEBUG [hbase-am-zkevent-worker-pool-2-thread-7] master.AssignmentManager$4(1189): The znode of region test,kkk,1369432820320.1ce9aafaaa422cef4ce0e1895111c733. has been deleted. 2013-05-24 22:00:38,495 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/region-in-transition 2013-05-24 22:00:38,495 INFO [hbase-am-zkevent-worker-pool-2-thread-7] master.RegionStates(264): Region {NAME => 'test,kkk,1369432820320.1ce9aafaaa422cef4ce0e1895111c733.', STARTKEY => 'kkk', ENDKEY => 'lll', ENCODED => 1ce9aafaaa422cef4ce0e1895111c733,} transitioned from {test,kkk,1369432820320.1ce9aafaaa422cef4ce0e1895111c733. state=OPEN, ts=1369432838025, server=kiyo.gq1.ygridcore.net,42690,1369432806911} to {test,kkk,1369432820320.1ce9aafaaa422cef4ce0e1895111c733. state=OPEN, ts=1369432838495, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:38,495 INFO [hbase-am-zkevent-worker-pool-2-thread-7] master.RegionStates(299): Onlined region {NAME => 'test,kkk,1369432820320.1ce9aafaaa422cef4ce0e1895111c733.', STARTKEY => 'kkk', ENDKEY => 'lll', ENCODED => 1ce9aafaaa422cef4ce0e1895111c733,} on kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:38,496 INFO [hbase-am-zkevent-worker-pool-2-thread-7] master.AssignmentManager$4(1194): The master has opened the region test,kkk,1369432820320.1ce9aafaaa422cef4ce0e1895111c733. that was online on kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:38,496 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.AssignmentManager(2753): Waiting on null to clear regions-in-transition 2013-05-24 22:00:38,496 DEBUG [MASTER_OPEN_REGION-kiyo.gq1.ygridcore.net,49101,1369432806564-1] zookeeper.ZKAssign(434): master:49101-0x13ed88e2cdf0004 Successfully deleted unassigned node for region 1ce9aafaaa422cef4ce0e1895111c733 in expected state RS_ZK_REGION_OPENED 2013-05-24 22:00:38,516 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/region-in-transition/ab7b9b9773243cff0ac09f3c379e33dc 2013-05-24 22:00:38,517 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] zookeeper.ZKAssign(862): regionserver:42690-0x13ed88e2cdf0006 Successfully transitioned node ab7b9b9773243cff0ac09f3c379e33dc from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-05-24 22:00:38,517 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] handler.OpenRegionHandler(373): region transitioned to opened in zookeeper: {NAME => 'test,mmm,1369432820320.ab7b9b9773243cff0ac09f3c379e33dc.', STARTKEY => 'mmm', ENDKEY => 'nnn', ENCODED => ab7b9b9773243cff0ac09f3c379e33dc,}, server: kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:38,517 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] handler.OpenRegionHandler(186): Opened test,mmm,1369432820320.ab7b9b9773243cff0ac09f3c379e33dc. on server:kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:38,517 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] zookeeper.ZKAssign(786): regionserver:42690-0x13ed88e2cdf0006 Attempting to transition node ec0b9e20bb8d4dc8fee9b3e80c5cae32 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-05-24 22:00:38,517 WARN [RegionServer:1;kiyo.gq1.ygridcore.net,39146,1369432814418] zookeeper.RecoverableZooKeeper(237): Possibly transient ZooKeeper, quorum=localhost:51411, exception=org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /2/rs/kiyo.gq1.ygridcore.net,39146,1369432814418 2013-05-24 22:00:38,518 INFO [RegionServer:1;kiyo.gq1.ygridcore.net,39146,1369432814418] util.RetryCounter(54): Sleeping 20ms before retry #1... 2013-05-24 22:00:38,520 INFO [PostOpenDeployTasks:0d6df81a73ad3b2a79add80716da62ac] regionserver.HRegionServer(1657): Post open deploy tasks for region=test,ooo,1369432820320.0d6df81a73ad3b2a79add80716da62ac. 2013-05-24 22:00:38,538 WARN [RegionServer:1;kiyo.gq1.ygridcore.net,39146,1369432814418] zookeeper.RecoverableZooKeeper(237): Possibly transient ZooKeeper, quorum=localhost:51411, exception=org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /2/rs/kiyo.gq1.ygridcore.net,39146,1369432814418 2013-05-24 22:00:38,538 ERROR [RegionServer:1;kiyo.gq1.ygridcore.net,39146,1369432814418] zookeeper.RecoverableZooKeeper(239): ZooKeeper delete failed after 1 retries 2013-05-24 22:00:38,538 WARN [RegionServer:1;kiyo.gq1.ygridcore.net,39146,1369432814418] regionserver.HRegionServer(935): Failed deleting my ephemeral node org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /2/rs/kiyo.gq1.ygridcore.net,39146,1369432814418 at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:152) at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1268) at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1257) at org.apache.hadoop.hbase.regionserver.HRegionServer.deleteMyEphemeralNode(HRegionServer.java:1193) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:933) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:158) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:110) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:142) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:337) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1457) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.util.Methods.call(Methods.java:41) at org.apache.hadoop.hbase.security.User.call(User.java:420) at org.apache.hadoop.hbase.security.User.access$300(User.java:51) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:260) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:140) at java.lang.Thread.run(Thread.java:662) 2013-05-24 22:00:38,539 INFO [RegionServer:1;kiyo.gq1.ygridcore.net,39146,1369432814418] regionserver.HRegionServer(941): stopping server kiyo.gq1.ygridcore.net,39146,1369432814418; zookeeper connection closed. 2013-05-24 22:00:38,539 INFO [RegionServer:1;kiyo.gq1.ygridcore.net,39146,1369432814418] regionserver.HRegionServer(944): RegionServer:1;kiyo.gq1.ygridcore.net,39146,1369432814418 exiting 2013-05-24 22:00:38,539 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@78d03350] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(193): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@78d03350 2013-05-24 22:00:38,566 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:51523-0x13ed88e2cdf0010 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/2/recovering-regions/45883ac87958f956040ff389c9f7eb86 2013-05-24 22:00:38,568 DEBUG [hbase-am-zkevent-worker-pool-2-thread-2] master.AssignmentManager(752): Handling transition=RS_ZK_REGION_OPENED, server=kiyo.gq1.ygridcore.net,42690,1369432806911, region=ab7b9b9773243cff0ac09f3c379e33dc, current state from region state map ={test,mmm,1369432820320.ab7b9b9773243cff0ac09f3c379e33dc. state=OPENING, ts=1369432837758, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:38,569 INFO [hbase-am-zkevent-worker-pool-2-thread-2] master.RegionStates(264): Region {NAME => 'test,mmm,1369432820320.ab7b9b9773243cff0ac09f3c379e33dc.', STARTKEY => 'mmm', ENDKEY => 'nnn', ENCODED => ab7b9b9773243cff0ac09f3c379e33dc,} transitioned from {test,mmm,1369432820320.ab7b9b9773243cff0ac09f3c379e33dc. state=OPENING, ts=1369432837758, server=kiyo.gq1.ygridcore.net,42690,1369432806911} to {test,mmm,1369432820320.ab7b9b9773243cff0ac09f3c379e33dc. state=OPEN, ts=1369432838569, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:38,569 DEBUG [MASTER_OPEN_REGION-kiyo.gq1.ygridcore.net,49101,1369432806564-2] handler.OpenedRegionHandler(146): Handling OPENED event for test,mmm,1369432820320.ab7b9b9773243cff0ac09f3c379e33dc. from kiyo.gq1.ygridcore.net,42690,1369432806911; deleting unassigned node 2013-05-24 22:00:38,569 DEBUG [MASTER_OPEN_REGION-kiyo.gq1.ygridcore.net,49101,1369432806564-2] zookeeper.ZKAssign(405): master:49101-0x13ed88e2cdf0004 Deleting existing unassigned node for ab7b9b9773243cff0ac09f3c379e33dc that is in expected state RS_ZK_REGION_OPENED 2013-05-24 22:00:38,601 DEBUG [PostOpenDeployTasks:45883ac87958f956040ff389c9f7eb86] regionserver.HRegionServer(4268): Update last flushed sequence id of region 45883ac87958f956040ff389c9f7eb86 for kiyo.gq1.ygridcore.net,39146,1369432814418 2013-05-24 22:00:38,679 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/region-in-transition/ec0b9e20bb8d4dc8fee9b3e80c5cae32 2013-05-24 22:00:38,680 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] zookeeper.ZKAssign(862): regionserver:42690-0x13ed88e2cdf0006 Successfully transitioned node ec0b9e20bb8d4dc8fee9b3e80c5cae32 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-05-24 22:00:38,680 INFO [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(4139): HRegion.openHRegion Region name ==test,qqq,1369432820320.ec0b9e20bb8d4dc8fee9b3e80c5cae32. 2013-05-24 22:00:38,680 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(4141): Opening region: {NAME => 'test,qqq,1369432820320.ec0b9e20bb8d4dc8fee9b3e80c5cae32.', STARTKEY => 'qqq', ENDKEY => 'rrr', ENCODED => ec0b9e20bb8d4dc8fee9b3e80c5cae32,} 2013-05-24 22:00:38,681 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.MetricsRegionSourceImpl(62): Creating new MetricsRegionSourceImpl for table test ec0b9e20bb8d4dc8fee9b3e80c5cae32 2013-05-24 22:00:38,681 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(518): Instantiated test,qqq,1369432820320.ec0b9e20bb8d4dc8fee9b3e80c5cae32. 2013-05-24 22:00:38,682 DEBUG [hbase-am-zkevent-worker-pool-2-thread-5] master.AssignmentManager(752): Handling transition=RS_ZK_REGION_OPENING, server=kiyo.gq1.ygridcore.net,42690,1369432806911, region=ec0b9e20bb8d4dc8fee9b3e80c5cae32, current state from region state map ={test,qqq,1369432820320.ec0b9e20bb8d4dc8fee9b3e80c5cae32. state=PENDING_OPEN, ts=1369432835960, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:38,682 INFO [hbase-am-zkevent-worker-pool-2-thread-5] master.RegionStates(264): Region {NAME => 'test,qqq,1369432820320.ec0b9e20bb8d4dc8fee9b3e80c5cae32.', STARTKEY => 'qqq', ENDKEY => 'rrr', ENCODED => ec0b9e20bb8d4dc8fee9b3e80c5cae32,} transitioned from {test,qqq,1369432820320.ec0b9e20bb8d4dc8fee9b3e80c5cae32. state=PENDING_OPEN, ts=1369432835960, server=kiyo.gq1.ygridcore.net,42690,1369432806911} to {test,qqq,1369432820320.ec0b9e20bb8d4dc8fee9b3e80c5cae32. state=OPENING, ts=1369432838682, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:38,683 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:42690-0x13ed88e2cdf0006 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/recovering-regions/53e3aef4cc8815dd31923659bff5ca21 2013-05-24 22:00:38,685 INFO [StoreOpenerThread-test,qqq,1369432820320.ec0b9e20bb8d4dc8fee9b3e80c5cae32.-1] compactions.CompactionConfiguration(85): Compaction configuration size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 604800000, major jitter 0.500000 2013-05-24 22:00:38,685 INFO [PostOpenDeployTasks:45883ac87958f956040ff389c9f7eb86] catalog.MetaEditor(432): Updated row test,,1369432825000.45883ac87958f956040ff389c9f7eb86. with server=kiyo.gq1.ygridcore.net,51523,1369432814392 2013-05-24 22:00:38,685 INFO [PostOpenDeployTasks:45883ac87958f956040ff389c9f7eb86] regionserver.HRegionServer(1682): Done with post open deploy task for region=test,,1369432825000.45883ac87958f956040ff389c9f7eb86. 2013-05-24 22:00:38,685 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,51523,1369432814392-0] zookeeper.ZKAssign(786): regionserver:51523-0x13ed88e2cdf0010 Attempting to transition node 45883ac87958f956040ff389c9f7eb86 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-05-24 22:00:38,687 INFO [StoreOpenerThread-test,qqq,1369432820320.ec0b9e20bb8d4dc8fee9b3e80c5cae32.-1] compactions.CompactionConfiguration(85): Compaction configuration size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 604800000, major jitter 0.500000 2013-05-24 22:00:38,690 INFO [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(608): Onlined test,qqq,1369432820320.ec0b9e20bb8d4dc8fee9b3e80c5cae32.; next sequenceid=1 2013-05-24 22:00:38,690 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] zookeeper.ZKAssign(598): regionserver:42690-0x13ed88e2cdf0006 Attempting to retransition the opening state of node ec0b9e20bb8d4dc8fee9b3e80c5cae32 2013-05-24 22:00:38,723 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/region-in-transition/ab7b9b9773243cff0ac09f3c379e33dc 2013-05-24 22:00:38,723 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/region-in-transition 2013-05-24 22:00:38,723 DEBUG [hbase-am-zkevent-worker-pool-2-thread-9] master.AssignmentManager$4(1189): The znode of region test,mmm,1369432820320.ab7b9b9773243cff0ac09f3c379e33dc. has been deleted. 2013-05-24 22:00:38,723 DEBUG [MASTER_OPEN_REGION-kiyo.gq1.ygridcore.net,49101,1369432806564-2] zookeeper.ZKAssign(434): master:49101-0x13ed88e2cdf0004 Successfully deleted unassigned node for region ab7b9b9773243cff0ac09f3c379e33dc in expected state RS_ZK_REGION_OPENED 2013-05-24 22:00:38,724 INFO [hbase-am-zkevent-worker-pool-2-thread-9] master.RegionStates(264): Region {NAME => 'test,mmm,1369432820320.ab7b9b9773243cff0ac09f3c379e33dc.', STARTKEY => 'mmm', ENDKEY => 'nnn', ENCODED => ab7b9b9773243cff0ac09f3c379e33dc,} transitioned from {test,mmm,1369432820320.ab7b9b9773243cff0ac09f3c379e33dc. state=OPEN, ts=1369432838569, server=kiyo.gq1.ygridcore.net,42690,1369432806911} to {test,mmm,1369432820320.ab7b9b9773243cff0ac09f3c379e33dc. state=OPEN, ts=1369432838724, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:38,724 INFO [hbase-am-zkevent-worker-pool-2-thread-9] master.RegionStates(299): Onlined region {NAME => 'test,mmm,1369432820320.ab7b9b9773243cff0ac09f3c379e33dc.', STARTKEY => 'mmm', ENDKEY => 'nnn', ENCODED => ab7b9b9773243cff0ac09f3c379e33dc,} on kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:38,724 INFO [hbase-am-zkevent-worker-pool-2-thread-9] master.AssignmentManager$4(1194): The master has opened the region test,mmm,1369432820320.ab7b9b9773243cff0ac09f3c379e33dc. that was online on kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:38,724 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.AssignmentManager(2753): Waiting on null to clear regions-in-transition 2013-05-24 22:00:38,782 DEBUG [PostOpenDeployTasks:53e3aef4cc8815dd31923659bff5ca21] regionserver.HRegionServer(4268): Update last flushed sequence id of region 53e3aef4cc8815dd31923659bff5ca21 for kiyo.gq1.ygridcore.net,49701,1369432806997 2013-05-24 22:00:38,784 INFO [PostOpenDeployTasks:ec0b9e20bb8d4dc8fee9b3e80c5cae32] regionserver.HRegionServer(1657): Post open deploy tasks for region=test,qqq,1369432820320.ec0b9e20bb8d4dc8fee9b3e80c5cae32. 2013-05-24 22:00:38,871 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:53632-0x13ed88e2cdf000f Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/2/region-in-transition/45883ac87958f956040ff389c9f7eb86 2013-05-24 22:00:38,871 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,51523,1369432814392-0] zookeeper.ZKAssign(862): regionserver:51523-0x13ed88e2cdf0010 Successfully transitioned node 45883ac87958f956040ff389c9f7eb86 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-05-24 22:00:38,872 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,51523,1369432814392-0] handler.OpenRegionHandler(373): region transitioned to opened in zookeeper: {NAME => 'test,,1369432825000.45883ac87958f956040ff389c9f7eb86.', STARTKEY => '', ENDKEY => '', ENCODED => 45883ac87958f956040ff389c9f7eb86,}, server: kiyo.gq1.ygridcore.net,51523,1369432814392 2013-05-24 22:00:38,872 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,51523,1369432814392-0] handler.OpenRegionHandler(186): Opened test,,1369432825000.45883ac87958f956040ff389c9f7eb86. on server:kiyo.gq1.ygridcore.net,51523,1369432814392 2013-05-24 22:00:38,939 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:42690-0x13ed88e2cdf0006 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/recovering-regions/0d6df81a73ad3b2a79add80716da62ac 2013-05-24 22:00:38,939 DEBUG [hbase-am-zkevent-worker-pool-13-thread-11] master.AssignmentManager(752): Handling transition=RS_ZK_REGION_OPENED, server=kiyo.gq1.ygridcore.net,51523,1369432814392, region=45883ac87958f956040ff389c9f7eb86, current state from region state map ={test,,1369432825000.45883ac87958f956040ff389c9f7eb86. state=OPENING, ts=1369432838022, server=kiyo.gq1.ygridcore.net,51523,1369432814392} 2013-05-24 22:00:38,940 INFO [hbase-am-zkevent-worker-pool-13-thread-11] master.RegionStates(264): Region {NAME => 'test,,1369432825000.45883ac87958f956040ff389c9f7eb86.', STARTKEY => '', ENDKEY => '', ENCODED => 45883ac87958f956040ff389c9f7eb86,} transitioned from {test,,1369432825000.45883ac87958f956040ff389c9f7eb86. state=OPENING, ts=1369432838022, server=kiyo.gq1.ygridcore.net,51523,1369432814392} to {test,,1369432825000.45883ac87958f956040ff389c9f7eb86. state=OPEN, ts=1369432838940, server=kiyo.gq1.ygridcore.net,51523,1369432814392} 2013-05-24 22:00:38,940 DEBUG [MASTER_OPEN_REGION-kiyo.gq1.ygridcore.net,53632,1369432814258-2] handler.OpenedRegionHandler(146): Handling OPENED event for test,,1369432825000.45883ac87958f956040ff389c9f7eb86. from kiyo.gq1.ygridcore.net,51523,1369432814392; deleting unassigned node 2013-05-24 22:00:38,940 DEBUG [MASTER_OPEN_REGION-kiyo.gq1.ygridcore.net,53632,1369432814258-2] zookeeper.ZKAssign(405): master:53632-0x13ed88e2cdf000f Deleting existing unassigned node for 45883ac87958f956040ff389c9f7eb86 that is in expected state RS_ZK_REGION_OPENED 2013-05-24 22:00:38,942 INFO [PostOpenDeployTasks:53e3aef4cc8815dd31923659bff5ca21] catalog.MetaEditor(432): Updated row test,nnn,1369432820320.53e3aef4cc8815dd31923659bff5ca21. with server=kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:38,942 INFO [PostOpenDeployTasks:53e3aef4cc8815dd31923659bff5ca21] regionserver.HRegionServer(1682): Done with post open deploy task for region=test,nnn,1369432820320.53e3aef4cc8815dd31923659bff5ca21. 2013-05-24 22:00:38,942 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] zookeeper.ZKAssign(786): regionserver:42690-0x13ed88e2cdf0006 Attempting to transition node 53e3aef4cc8815dd31923659bff5ca21 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-05-24 22:00:38,996 DEBUG [PostOpenDeployTasks:0d6df81a73ad3b2a79add80716da62ac] regionserver.HRegionServer(4268): Update last flushed sequence id of region 0d6df81a73ad3b2a79add80716da62ac for kiyo.gq1.ygridcore.net,49701,1369432806997 2013-05-24 22:00:38,997 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432834215 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:39,001 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:39,004 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(583): Nothing to replicate, sleeping 100 times 9 2013-05-24 22:00:39,017 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:53632-0x13ed88e2cdf000f Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/2/region-in-transition/45883ac87958f956040ff389c9f7eb86 2013-05-24 22:00:39,018 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:53632-0x13ed88e2cdf000f Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/2/region-in-transition 2013-05-24 22:00:39,018 DEBUG [hbase-am-zkevent-worker-pool-13-thread-12] master.AssignmentManager$4(1189): The znode of region test,,1369432825000.45883ac87958f956040ff389c9f7eb86. has been deleted. 2013-05-24 22:00:39,018 INFO [hbase-am-zkevent-worker-pool-13-thread-12] master.RegionStates(264): Region {NAME => 'test,,1369432825000.45883ac87958f956040ff389c9f7eb86.', STARTKEY => '', ENDKEY => '', ENCODED => 45883ac87958f956040ff389c9f7eb86,} transitioned from {test,,1369432825000.45883ac87958f956040ff389c9f7eb86. state=OPEN, ts=1369432838940, server=kiyo.gq1.ygridcore.net,51523,1369432814392} to {test,,1369432825000.45883ac87958f956040ff389c9f7eb86. state=OPEN, ts=1369432839018, server=kiyo.gq1.ygridcore.net,51523,1369432814392} 2013-05-24 22:00:39,018 DEBUG [MASTER_OPEN_REGION-kiyo.gq1.ygridcore.net,53632,1369432814258-2] zookeeper.ZKAssign(434): master:53632-0x13ed88e2cdf000f Successfully deleted unassigned node for region 45883ac87958f956040ff389c9f7eb86 in expected state RS_ZK_REGION_OPENED 2013-05-24 22:00:39,018 INFO [hbase-am-zkevent-worker-pool-13-thread-12] master.RegionStates(299): Onlined region {NAME => 'test,,1369432825000.45883ac87958f956040ff389c9f7eb86.', STARTKEY => '', ENDKEY => '', ENCODED => 45883ac87958f956040ff389c9f7eb86,} on kiyo.gq1.ygridcore.net,51523,1369432814392 2013-05-24 22:00:39,018 INFO [hbase-am-zkevent-worker-pool-13-thread-12] master.AssignmentManager$4(1194): The master has opened the region test,,1369432825000.45883ac87958f956040ff389c9f7eb86. that was online on kiyo.gq1.ygridcore.net,51523,1369432814392 2013-05-24 22:00:39,055 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/region-in-transition/53e3aef4cc8815dd31923659bff5ca21 2013-05-24 22:00:39,057 INFO [PostOpenDeployTasks:0d6df81a73ad3b2a79add80716da62ac] catalog.MetaEditor(432): Updated row test,ooo,1369432820320.0d6df81a73ad3b2a79add80716da62ac. with server=kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:39,057 DEBUG [hbase-am-zkevent-worker-pool-2-thread-6] master.AssignmentManager(752): Handling transition=RS_ZK_REGION_OPENED, server=kiyo.gq1.ygridcore.net,42690,1369432806911, region=53e3aef4cc8815dd31923659bff5ca21, current state from region state map ={test,nnn,1369432820320.53e3aef4cc8815dd31923659bff5ca21. state=OPENING, ts=1369432838026, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:39,057 INFO [PostOpenDeployTasks:0d6df81a73ad3b2a79add80716da62ac] regionserver.HRegionServer(1682): Done with post open deploy task for region=test,ooo,1369432820320.0d6df81a73ad3b2a79add80716da62ac. 2013-05-24 22:00:39,057 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] zookeeper.ZKAssign(862): regionserver:42690-0x13ed88e2cdf0006 Successfully transitioned node 53e3aef4cc8815dd31923659bff5ca21 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-05-24 22:00:39,057 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] zookeeper.ZKAssign(786): regionserver:42690-0x13ed88e2cdf0006 Attempting to transition node 0d6df81a73ad3b2a79add80716da62ac from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-05-24 22:00:39,057 INFO [hbase-am-zkevent-worker-pool-2-thread-6] master.RegionStates(264): Region {NAME => 'test,nnn,1369432820320.53e3aef4cc8815dd31923659bff5ca21.', STARTKEY => 'nnn', ENDKEY => 'ooo', ENCODED => 53e3aef4cc8815dd31923659bff5ca21,} transitioned from {test,nnn,1369432820320.53e3aef4cc8815dd31923659bff5ca21. state=OPENING, ts=1369432838026, server=kiyo.gq1.ygridcore.net,42690,1369432806911} to {test,nnn,1369432820320.53e3aef4cc8815dd31923659bff5ca21. state=OPEN, ts=1369432839057, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:39,058 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] handler.OpenRegionHandler(373): region transitioned to opened in zookeeper: {NAME => 'test,nnn,1369432820320.53e3aef4cc8815dd31923659bff5ca21.', STARTKEY => 'nnn', ENDKEY => 'ooo', ENCODED => 53e3aef4cc8815dd31923659bff5ca21,}, server: kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:39,058 DEBUG [MASTER_OPEN_REGION-kiyo.gq1.ygridcore.net,49101,1369432806564-3] handler.OpenedRegionHandler(146): Handling OPENED event for test,nnn,1369432820320.53e3aef4cc8815dd31923659bff5ca21. from kiyo.gq1.ygridcore.net,42690,1369432806911; deleting unassigned node 2013-05-24 22:00:39,058 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] handler.OpenRegionHandler(186): Opened test,nnn,1369432820320.53e3aef4cc8815dd31923659bff5ca21. on server:kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:39,058 DEBUG [MASTER_OPEN_REGION-kiyo.gq1.ygridcore.net,49101,1369432806564-3] zookeeper.ZKAssign(405): master:49101-0x13ed88e2cdf0004 Deleting existing unassigned node for 53e3aef4cc8815dd31923659bff5ca21 that is in expected state RS_ZK_REGION_OPENED 2013-05-24 22:00:39,058 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] zookeeper.ZKAssign(786): regionserver:42690-0x13ed88e2cdf0006 Attempting to transition node 872960a61a4ceeb9f64577d8b7cb0ddf from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-05-24 22:00:39,064 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,53632,1369432814258-0] master.MasterFileSystem(322): Renamed region directory: hdfs://localhost:54104/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,39146,1369432814418-splitting 2013-05-24 22:00:39,065 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,53632,1369432814258-0] master.SplitLogManager(1283): dead splitlog workers [kiyo.gq1.ygridcore.net,39146,1369432814418] 2013-05-24 22:00:39,068 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,53632,1369432814258-0] master.SplitLogManager(304): Scheduling batch of logs to split 2013-05-24 22:00:39,070 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,53632,1369432814258-0] master.SplitLogManager(306): started splitting 6 logs in [hdfs://localhost:54104/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,39146,1369432814418-splitting] 2013-05-24 22:00:39,105 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:42690-0x13ed88e2cdf0006 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/recovering-regions/ec0b9e20bb8d4dc8fee9b3e80c5cae32 2013-05-24 22:00:39,158 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:51523-0x13ed88e2cdf0010 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/2/splitlog 2013-05-24 22:00:39,158 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392-EventThread] regionserver.SplitLogWorker(572): tasks arrived or departed 2013-05-24 22:00:39,159 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(666): put up splitlog task at znode /2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432818197 2013-05-24 22:00:39,160 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(666): put up splitlog task at znode /2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432828321 2013-05-24 22:00:39,160 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(666): put up splitlog task at znode /2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432829746 2013-05-24 22:00:39,161 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(666): put up splitlog task at znode /2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432831597 2013-05-24 22:00:39,162 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(666): put up splitlog task at znode /2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432832288 2013-05-24 22:00:39,163 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(666): put up splitlog task at znode /2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432833356 2013-05-24 22:00:39,212 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/region-in-transition/0d6df81a73ad3b2a79add80716da62ac 2013-05-24 22:00:39,212 DEBUG [PostOpenDeployTasks:ec0b9e20bb8d4dc8fee9b3e80c5cae32] regionserver.HRegionServer(4268): Update last flushed sequence id of region ec0b9e20bb8d4dc8fee9b3e80c5cae32 for kiyo.gq1.ygridcore.net,49701,1369432806997 2013-05-24 22:00:39,213 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] zookeeper.ZKAssign(862): regionserver:42690-0x13ed88e2cdf0006 Successfully transitioned node 0d6df81a73ad3b2a79add80716da62ac from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-05-24 22:00:39,214 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] handler.OpenRegionHandler(373): region transitioned to opened in zookeeper: {NAME => 'test,ooo,1369432820320.0d6df81a73ad3b2a79add80716da62ac.', STARTKEY => 'ooo', ENDKEY => 'ppp', ENCODED => 0d6df81a73ad3b2a79add80716da62ac,}, server: kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:39,214 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] handler.OpenRegionHandler(186): Opened test,ooo,1369432820320.0d6df81a73ad3b2a79add80716da62ac. on server:kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:39,214 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] zookeeper.ZKAssign(786): regionserver:42690-0x13ed88e2cdf0006 Attempting to transition node dba8b67ade91438cad8a5209736330e6 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-05-24 22:00:39,234 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/region-in-transition/53e3aef4cc8815dd31923659bff5ca21 2013-05-24 22:00:39,234 DEBUG [hbase-am-zkevent-worker-pool-2-thread-12] master.AssignmentManager$4(1189): The znode of region test,nnn,1369432820320.53e3aef4cc8815dd31923659bff5ca21. has been deleted. 2013-05-24 22:00:39,235 INFO [hbase-am-zkevent-worker-pool-2-thread-12] master.RegionStates(264): Region {NAME => 'test,nnn,1369432820320.53e3aef4cc8815dd31923659bff5ca21.', STARTKEY => 'nnn', ENDKEY => 'ooo', ENCODED => 53e3aef4cc8815dd31923659bff5ca21,} transitioned from {test,nnn,1369432820320.53e3aef4cc8815dd31923659bff5ca21. state=OPEN, ts=1369432839057, server=kiyo.gq1.ygridcore.net,42690,1369432806911} to {test,nnn,1369432820320.53e3aef4cc8815dd31923659bff5ca21. state=OPEN, ts=1369432839235, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:39,235 INFO [hbase-am-zkevent-worker-pool-2-thread-12] master.RegionStates(299): Onlined region {NAME => 'test,nnn,1369432820320.53e3aef4cc8815dd31923659bff5ca21.', STARTKEY => 'nnn', ENDKEY => 'ooo', ENCODED => 53e3aef4cc8815dd31923659bff5ca21,} on kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:39,235 INFO [hbase-am-zkevent-worker-pool-2-thread-12] master.AssignmentManager$4(1194): The master has opened the region test,nnn,1369432820320.53e3aef4cc8815dd31923659bff5ca21. that was online on kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:39,235 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.AssignmentManager(2753): Waiting on null to clear regions-in-transition 2013-05-24 22:00:39,235 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] zookeeper.ZKAssign(862): regionserver:42690-0x13ed88e2cdf0006 Successfully transitioned node 872960a61a4ceeb9f64577d8b7cb0ddf from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-05-24 22:00:39,235 INFO [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegion(4139): HRegion.openHRegion Region name ==test,sss,1369432820320.872960a61a4ceeb9f64577d8b7cb0ddf. 2013-05-24 22:00:39,235 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegion(4141): Opening region: {NAME => 'test,sss,1369432820320.872960a61a4ceeb9f64577d8b7cb0ddf.', STARTKEY => 'sss', ENDKEY => 'ttt', ENCODED => 872960a61a4ceeb9f64577d8b7cb0ddf,} 2013-05-24 22:00:39,236 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(708): task not yet acquired /2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432818197 ver = 0 2013-05-24 22:00:39,236 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.MetricsRegionSourceImpl(62): Creating new MetricsRegionSourceImpl for table test 872960a61a4ceeb9f64577d8b7cb0ddf 2013-05-24 22:00:39,236 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegion(518): Instantiated test,sss,1369432820320.872960a61a4ceeb9f64577d8b7cb0ddf. 2013-05-24 22:00:39,236 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/region-in-transition 2013-05-24 22:00:39,237 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(708): task not yet acquired /2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432828321 ver = 0 2013-05-24 22:00:39,237 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(708): task not yet acquired /2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432829746 ver = 0 2013-05-24 22:00:39,238 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(708): task not yet acquired /2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432831597 ver = 0 2013-05-24 22:00:39,239 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(708): task not yet acquired /2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432832288 ver = 0 2013-05-24 22:00:39,240 DEBUG [MASTER_OPEN_REGION-kiyo.gq1.ygridcore.net,49101,1369432806564-3] zookeeper.ZKAssign(434): master:49101-0x13ed88e2cdf0004 Successfully deleted unassigned node for region 53e3aef4cc8815dd31923659bff5ca21 in expected state RS_ZK_REGION_OPENED 2013-05-24 22:00:39,240 INFO [StoreOpenerThread-test,sss,1369432820320.872960a61a4ceeb9f64577d8b7cb0ddf.-1] compactions.CompactionConfiguration(85): Compaction configuration size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 604800000, major jitter 0.500000 2013-05-24 22:00:39,240 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(708): task not yet acquired /2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432833356 ver = 0 2013-05-24 22:00:39,241 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/region-in-transition/872960a61a4ceeb9f64577d8b7cb0ddf 2013-05-24 22:00:39,242 DEBUG [hbase-am-zkevent-worker-pool-2-thread-10] master.AssignmentManager(752): Handling transition=RS_ZK_REGION_OPENED, server=kiyo.gq1.ygridcore.net,42690,1369432806911, region=0d6df81a73ad3b2a79add80716da62ac, current state from region state map ={test,ooo,1369432820320.0d6df81a73ad3b2a79add80716da62ac. state=OPENING, ts=1369432838356, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:39,242 INFO [hbase-am-zkevent-worker-pool-2-thread-10] master.RegionStates(264): Region {NAME => 'test,ooo,1369432820320.0d6df81a73ad3b2a79add80716da62ac.', STARTKEY => 'ooo', ENDKEY => 'ppp', ENCODED => 0d6df81a73ad3b2a79add80716da62ac,} transitioned from {test,ooo,1369432820320.0d6df81a73ad3b2a79add80716da62ac. state=OPENING, ts=1369432838356, server=kiyo.gq1.ygridcore.net,42690,1369432806911} to {test,ooo,1369432820320.0d6df81a73ad3b2a79add80716da62ac. state=OPEN, ts=1369432839242, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:39,242 INFO [StoreOpenerThread-test,sss,1369432820320.872960a61a4ceeb9f64577d8b7cb0ddf.-1] compactions.CompactionConfiguration(85): Compaction configuration size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 604800000, major jitter 0.500000 2013-05-24 22:00:39,242 DEBUG [MASTER_OPEN_REGION-kiyo.gq1.ygridcore.net,49101,1369432806564-4] handler.OpenedRegionHandler(146): Handling OPENED event for test,ooo,1369432820320.0d6df81a73ad3b2a79add80716da62ac. from kiyo.gq1.ygridcore.net,42690,1369432806911; deleting unassigned node 2013-05-24 22:00:39,242 DEBUG [MASTER_OPEN_REGION-kiyo.gq1.ygridcore.net,49101,1369432806564-4] zookeeper.ZKAssign(405): master:49101-0x13ed88e2cdf0004 Deleting existing unassigned node for 0d6df81a73ad3b2a79add80716da62ac that is in expected state RS_ZK_REGION_OPENED 2013-05-24 22:00:39,245 INFO [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegion(608): Onlined test,sss,1369432820320.872960a61a4ceeb9f64577d8b7cb0ddf.; next sequenceid=1 2013-05-24 22:00:39,245 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] zookeeper.ZKAssign(598): regionserver:42690-0x13ed88e2cdf0006 Attempting to retransition the opening state of node 872960a61a4ceeb9f64577d8b7cb0ddf 2013-05-24 22:00:39,250 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/region-in-transition/dba8b67ade91438cad8a5209736330e6 2013-05-24 22:00:39,251 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:53632-0x13ed88e2cdf000f Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432829746 2013-05-24 22:00:39,251 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] zookeeper.ZKAssign(862): regionserver:42690-0x13ed88e2cdf0006 Successfully transitioned node dba8b67ade91438cad8a5209736330e6 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-05-24 22:00:39,251 INFO [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegion(4139): HRegion.openHRegion Region name ==test,uuu,1369432820320.dba8b67ade91438cad8a5209736330e6. 2013-05-24 22:00:39,251 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegion(4141): Opening region: {NAME => 'test,uuu,1369432820320.dba8b67ade91438cad8a5209736330e6.', STARTKEY => 'uuu', ENDKEY => 'vvv', ENCODED => dba8b67ade91438cad8a5209736330e6,} 2013-05-24 22:00:39,252 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.MetricsRegionSourceImpl(62): Creating new MetricsRegionSourceImpl for table test dba8b67ade91438cad8a5209736330e6 2013-05-24 22:00:39,252 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegion(518): Instantiated test,uuu,1369432820320.dba8b67ade91438cad8a5209736330e6. 2013-05-24 22:00:39,252 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] regionserver.SplitLogWorker(330): worker kiyo.gq1.ygridcore.net,51523,1369432814392 acquired task /2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432829746 2013-05-24 22:00:39,253 INFO [PostOpenDeployTasks:ec0b9e20bb8d4dc8fee9b3e80c5cae32] catalog.MetaEditor(432): Updated row test,qqq,1369432820320.ec0b9e20bb8d4dc8fee9b3e80c5cae32. with server=kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:39,253 INFO [PostOpenDeployTasks:ec0b9e20bb8d4dc8fee9b3e80c5cae32] regionserver.HRegionServer(1682): Done with post open deploy task for region=test,qqq,1369432820320.ec0b9e20bb8d4dc8fee9b3e80c5cae32. 2013-05-24 22:00:39,253 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] zookeeper.ZKAssign(786): regionserver:42690-0x13ed88e2cdf0006 Attempting to transition node ec0b9e20bb8d4dc8fee9b3e80c5cae32 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-05-24 22:00:39,254 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(781): task /2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432829746 acquired by kiyo.gq1.ygridcore.net,51523,1369432814392 2013-05-24 22:00:39,253 DEBUG [hbase-am-zkevent-worker-pool-2-thread-8] master.AssignmentManager(752): Handling transition=RS_ZK_REGION_OPENING, server=kiyo.gq1.ygridcore.net,42690,1369432806911, region=872960a61a4ceeb9f64577d8b7cb0ddf, current state from region state map ={test,sss,1369432820320.872960a61a4ceeb9f64577d8b7cb0ddf. state=PENDING_OPEN, ts=1369432835960, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:39,254 INFO [hbase-am-zkevent-worker-pool-2-thread-8] master.RegionStates(264): Region {NAME => 'test,sss,1369432820320.872960a61a4ceeb9f64577d8b7cb0ddf.', STARTKEY => 'sss', ENDKEY => 'ttt', ENCODED => 872960a61a4ceeb9f64577d8b7cb0ddf,} transitioned from {test,sss,1369432820320.872960a61a4ceeb9f64577d8b7cb0ddf. state=PENDING_OPEN, ts=1369432835960, server=kiyo.gq1.ygridcore.net,42690,1369432806911} to {test,sss,1369432820320.872960a61a4ceeb9f64577d8b7cb0ddf. state=OPENING, ts=1369432839254, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:39,255 DEBUG [hbase-am-zkevent-worker-pool-2-thread-16] master.AssignmentManager(752): Handling transition=RS_ZK_REGION_OPENING, server=kiyo.gq1.ygridcore.net,42690,1369432806911, region=dba8b67ade91438cad8a5209736330e6, current state from region state map ={test,uuu,1369432820320.dba8b67ade91438cad8a5209736330e6. state=PENDING_OPEN, ts=1369432835960, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:39,255 INFO [StoreOpenerThread-test,uuu,1369432820320.dba8b67ade91438cad8a5209736330e6.-1] compactions.CompactionConfiguration(85): Compaction configuration size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 604800000, major jitter 0.500000 2013-05-24 22:00:39,255 INFO [hbase-am-zkevent-worker-pool-2-thread-16] master.RegionStates(264): Region {NAME => 'test,uuu,1369432820320.dba8b67ade91438cad8a5209736330e6.', STARTKEY => 'uuu', ENDKEY => 'vvv', ENCODED => dba8b67ade91438cad8a5209736330e6,} transitioned from {test,uuu,1369432820320.dba8b67ade91438cad8a5209736330e6. state=PENDING_OPEN, ts=1369432835960, server=kiyo.gq1.ygridcore.net,42690,1369432806911} to {test,uuu,1369432820320.dba8b67ade91438cad8a5209736330e6. state=OPENING, ts=1369432839255, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:39,256 INFO [PostOpenDeployTasks:872960a61a4ceeb9f64577d8b7cb0ddf] regionserver.HRegionServer(1657): Post open deploy tasks for region=test,sss,1369432820320.872960a61a4ceeb9f64577d8b7cb0ddf. 2013-05-24 22:00:39,258 INFO [StoreOpenerThread-test,uuu,1369432820320.dba8b67ade91438cad8a5209736330e6.-1] compactions.CompactionConfiguration(85): Compaction configuration size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 604800000, major jitter 0.500000 2013-05-24 22:00:39,261 INFO [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegion(608): Onlined test,uuu,1369432820320.dba8b67ade91438cad8a5209736330e6.; next sequenceid=1 2013-05-24 22:00:39,261 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] zookeeper.ZKAssign(598): regionserver:42690-0x13ed88e2cdf0006 Attempting to retransition the opening state of node dba8b67ade91438cad8a5209736330e6 2013-05-24 22:00:39,261 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/region-in-transition/0d6df81a73ad3b2a79add80716da62ac 2013-05-24 22:00:39,261 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] wal.HLogSplitter(506): Splitting hlog: hdfs://localhost:54104/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,39146,1369432814418-splitting/kiyo.gq1.ygridcore.net%2C39146%2C1369432814418.1369432829746, length=43873 2013-05-24 22:00:39,261 DEBUG [hbase-am-zkevent-worker-pool-2-thread-13] master.AssignmentManager$4(1189): The znode of region test,ooo,1369432820320.0d6df81a73ad3b2a79add80716da62ac. has been deleted. 2013-05-24 22:00:39,261 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/region-in-transition 2013-05-24 22:00:39,262 INFO [hbase-am-zkevent-worker-pool-2-thread-13] master.RegionStates(264): Region {NAME => 'test,ooo,1369432820320.0d6df81a73ad3b2a79add80716da62ac.', STARTKEY => 'ooo', ENDKEY => 'ppp', ENCODED => 0d6df81a73ad3b2a79add80716da62ac,} transitioned from {test,ooo,1369432820320.0d6df81a73ad3b2a79add80716da62ac. state=OPEN, ts=1369432839242, server=kiyo.gq1.ygridcore.net,42690,1369432806911} to {test,ooo,1369432820320.0d6df81a73ad3b2a79add80716da62ac. state=OPEN, ts=1369432839262, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:39,261 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] wal.HLogSplitter(507): DistributedLogReplay = true 2013-05-24 22:00:39,262 DEBUG [MASTER_OPEN_REGION-kiyo.gq1.ygridcore.net,49101,1369432806564-4] zookeeper.ZKAssign(434): master:49101-0x13ed88e2cdf0004 Successfully deleted unassigned node for region 0d6df81a73ad3b2a79add80716da62ac in expected state RS_ZK_REGION_OPENED 2013-05-24 22:00:39,262 INFO [hbase-am-zkevent-worker-pool-2-thread-13] master.RegionStates(299): Onlined region {NAME => 'test,ooo,1369432820320.0d6df81a73ad3b2a79add80716da62ac.', STARTKEY => 'ooo', ENDKEY => 'ppp', ENCODED => 0d6df81a73ad3b2a79add80716da62ac,} on kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:39,262 INFO [hbase-am-zkevent-worker-pool-2-thread-13] master.AssignmentManager$4(1194): The master has opened the region test,ooo,1369432820320.0d6df81a73ad3b2a79add80716da62ac. that was online on kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:39,262 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.AssignmentManager(2753): Waiting on null to clear regions-in-transition 2013-05-24 22:00:39,269 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:42690-0x13ed88e2cdf0006 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/replication/rs/kiyo.gq1.ygridcore.net,49701,1369432806997/lock 2013-05-24 22:00:39,272 INFO [PostOpenDeployTasks:dba8b67ade91438cad8a5209736330e6] regionserver.HRegionServer(1657): Post open deploy tasks for region=test,uuu,1369432820320.dba8b67ade91438cad8a5209736330e6. 2013-05-24 22:00:39,278 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:51523-0x13ed88e2cdf0010 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432829746 2013-05-24 22:00:39,278 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:53632-0x13ed88e2cdf000f Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432829746 2013-05-24 22:00:39,279 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] util.FSHDFSUtils(86): Recovering lease on dfs file hdfs://localhost:54104/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,39146,1369432814418-splitting/kiyo.gq1.ygridcore.net%2C39146%2C1369432814418.1369432829746 2013-05-24 22:00:39,281 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] util.FSHDFSUtils(156): recoverLease=true, attempt=0 on file=hdfs://localhost:54104/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,39146,1369432814418-splitting/kiyo.gq1.ygridcore.net%2C39146%2C1369432814418.1369432829746 after 1ms 2013-05-24 22:00:39,285 DEBUG [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 43865, fileLength: 43873, trailerPresent: true 2013-05-24 22:00:39,286 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/region-in-transition/ec0b9e20bb8d4dc8fee9b3e80c5cae32 2013-05-24 22:00:39,288 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] zookeeper.ZKAssign(862): regionserver:42690-0x13ed88e2cdf0006 Successfully transitioned node ec0b9e20bb8d4dc8fee9b3e80c5cae32 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-05-24 22:00:39,288 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] handler.OpenRegionHandler(373): region transitioned to opened in zookeeper: {NAME => 'test,qqq,1369432820320.ec0b9e20bb8d4dc8fee9b3e80c5cae32.', STARTKEY => 'qqq', ENDKEY => 'rrr', ENCODED => ec0b9e20bb8d4dc8fee9b3e80c5cae32,}, server: kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:39,289 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] handler.OpenRegionHandler(186): Opened test,qqq,1369432820320.ec0b9e20bb8d4dc8fee9b3e80c5cae32. on server:kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:39,289 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] zookeeper.ZKAssign(786): regionserver:42690-0x13ed88e2cdf0006 Attempting to transition node acf78311e603cb0d02309ae08763d196 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-05-24 22:00:39,289 DEBUG [hbase-am-zkevent-worker-pool-2-thread-17] master.AssignmentManager(752): Handling transition=RS_ZK_REGION_OPENED, server=kiyo.gq1.ygridcore.net,42690,1369432806911, region=ec0b9e20bb8d4dc8fee9b3e80c5cae32, current state from region state map ={test,qqq,1369432820320.ec0b9e20bb8d4dc8fee9b3e80c5cae32. state=OPENING, ts=1369432838682, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:39,289 INFO [hbase-am-zkevent-worker-pool-2-thread-17] master.RegionStates(264): Region {NAME => 'test,qqq,1369432820320.ec0b9e20bb8d4dc8fee9b3e80c5cae32.', STARTKEY => 'qqq', ENDKEY => 'rrr', ENCODED => ec0b9e20bb8d4dc8fee9b3e80c5cae32,} transitioned from {test,qqq,1369432820320.ec0b9e20bb8d4dc8fee9b3e80c5cae32. state=OPENING, ts=1369432838682, server=kiyo.gq1.ygridcore.net,42690,1369432806911} to {test,qqq,1369432820320.ec0b9e20bb8d4dc8fee9b3e80c5cae32. state=OPEN, ts=1369432839289, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:39,290 DEBUG [MASTER_OPEN_REGION-kiyo.gq1.ygridcore.net,49101,1369432806564-0] handler.OpenedRegionHandler(146): Handling OPENED event for test,qqq,1369432820320.ec0b9e20bb8d4dc8fee9b3e80c5cae32. from kiyo.gq1.ygridcore.net,42690,1369432806911; deleting unassigned node 2013-05-24 22:00:39,290 DEBUG [MASTER_OPEN_REGION-kiyo.gq1.ygridcore.net,49101,1369432806564-0] zookeeper.ZKAssign(405): master:49101-0x13ed88e2cdf0004 Deleting existing unassigned node for ec0b9e20bb8d4dc8fee9b3e80c5cae32 that is in expected state RS_ZK_REGION_OPENED 2013-05-24 22:00:39,293 DEBUG [WriterThread-1] wal.HLogSplitter$WriterThread(1102): Writer thread Thread[WriterThread-1,5,main]: starting 2013-05-24 22:00:39,293 DEBUG [WriterThread-0] wal.HLogSplitter$WriterThread(1102): Writer thread Thread[WriterThread-0,5,main]: starting 2013-05-24 22:00:39,293 DEBUG [WriterThread-2] wal.HLogSplitter$WriterThread(1102): Writer thread Thread[WriterThread-2,5,main]: starting 2013-05-24 22:00:39,296 DEBUG [ReplicationExecutor-0] replication.ReplicationQueueInfo(109): Found dead servers:[kiyo.gq1.ygridcore.net,49701,1369432806997] 2013-05-24 22:00:39,303 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/region-in-transition/acf78311e603cb0d02309ae08763d196 2013-05-24 22:00:39,303 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] zookeeper.ZKAssign(862): regionserver:42690-0x13ed88e2cdf0006 Successfully transitioned node acf78311e603cb0d02309ae08763d196 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-05-24 22:00:39,303 INFO [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(4139): HRegion.openHRegion Region name ==test,yyy,1369432820321.acf78311e603cb0d02309ae08763d196. 2013-05-24 22:00:39,303 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(4141): Opening region: {NAME => 'test,yyy,1369432820321.acf78311e603cb0d02309ae08763d196.', STARTKEY => 'yyy', ENDKEY => 'zzz', ENCODED => acf78311e603cb0d02309ae08763d196,} 2013-05-24 22:00:39,304 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.MetricsRegionSourceImpl(62): Creating new MetricsRegionSourceImpl for table test acf78311e603cb0d02309ae08763d196 2013-05-24 22:00:39,304 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(518): Instantiated test,yyy,1369432820321.acf78311e603cb0d02309ae08763d196. 2013-05-24 22:00:39,308 INFO [StoreOpenerThread-test,yyy,1369432820321.acf78311e603cb0d02309ae08763d196.-1] compactions.CompactionConfiguration(85): Compaction configuration size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 604800000, major jitter 0.500000 2013-05-24 22:00:39,310 INFO [StoreOpenerThread-test,yyy,1369432820321.acf78311e603cb0d02309ae08763d196.-1] compactions.CompactionConfiguration(85): Compaction configuration size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 604800000, major jitter 0.500000 2013-05-24 22:00:39,311 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/region-in-transition/ec0b9e20bb8d4dc8fee9b3e80c5cae32 2013-05-24 22:00:39,311 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:42690-0x13ed88e2cdf0006 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/recovering-regions/872960a61a4ceeb9f64577d8b7cb0ddf 2013-05-24 22:00:39,311 DEBUG [hbase-am-zkevent-worker-pool-2-thread-18] master.AssignmentManager$4(1189): The znode of region test,qqq,1369432820320.ec0b9e20bb8d4dc8fee9b3e80c5cae32. has been deleted. 2013-05-24 22:00:39,312 INFO [hbase-am-zkevent-worker-pool-2-thread-18] master.RegionStates(264): Region {NAME => 'test,qqq,1369432820320.ec0b9e20bb8d4dc8fee9b3e80c5cae32.', STARTKEY => 'qqq', ENDKEY => 'rrr', ENCODED => ec0b9e20bb8d4dc8fee9b3e80c5cae32,} transitioned from {test,qqq,1369432820320.ec0b9e20bb8d4dc8fee9b3e80c5cae32. state=OPEN, ts=1369432839289, server=kiyo.gq1.ygridcore.net,42690,1369432806911} to {test,qqq,1369432820320.ec0b9e20bb8d4dc8fee9b3e80c5cae32. state=OPEN, ts=1369432839311, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:39,312 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/region-in-transition 2013-05-24 22:00:39,312 INFO [ReplicationExecutor-0.replicationSource,2-kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.ReplicationSource(211): Getting 1 rs from peer cluster # 2 2013-05-24 22:00:39,312 INFO [hbase-am-zkevent-worker-pool-2-thread-18] master.RegionStates(299): Onlined region {NAME => 'test,qqq,1369432820320.ec0b9e20bb8d4dc8fee9b3e80c5cae32.', STARTKEY => 'qqq', ENDKEY => 'rrr', ENCODED => ec0b9e20bb8d4dc8fee9b3e80c5cae32,} on kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:39,312 INFO [ReplicationExecutor-0.replicationSource,2-kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.ReplicationSource(219): Choosing peer kiyo.gq1.ygridcore.net,51523,1369432814392 2013-05-24 22:00:39,312 INFO [hbase-am-zkevent-worker-pool-2-thread-18] master.AssignmentManager$4(1194): The master has opened the region test,qqq,1369432820320.ec0b9e20bb8d4dc8fee9b3e80c5cae32. that was online on kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:39,312 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.AssignmentManager(2753): Waiting on null to clear regions-in-transition 2013-05-24 22:00:39,313 DEBUG [MASTER_OPEN_REGION-kiyo.gq1.ygridcore.net,49101,1369432806564-0] zookeeper.ZKAssign(434): master:49101-0x13ed88e2cdf0004 Successfully deleted unassigned node for region ec0b9e20bb8d4dc8fee9b3e80c5cae32 in expected state RS_ZK_REGION_OPENED 2013-05-24 22:00:39,314 INFO [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(608): Onlined test,yyy,1369432820321.acf78311e603cb0d02309ae08763d196.; next sequenceid=1 2013-05-24 22:00:39,314 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] zookeeper.ZKAssign(598): regionserver:42690-0x13ed88e2cdf0006 Attempting to retransition the opening state of node acf78311e603cb0d02309ae08763d196 2013-05-24 22:00:39,314 INFO [ReplicationExecutor-0.replicationSource,2-kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.ReplicationSource(252): Replicating 9df804a9-4d74-4edc-b420-4f9ebca9e588 -> 6b1ff12c-b39a-4d1a-9815-8c207023a768 2013-05-24 22:00:39,314 INFO [WriterThread-2] zookeeper.RecoverableZooKeeper(119): Process identifier=hconnection-0x43f1a352 connecting to ZooKeeper ensemble=localhost:51411 2013-05-24 22:00:39,315 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] wal.HLogSplitter(595): Finishing writing output logs and closing down. 2013-05-24 22:00:39,315 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] wal.HLogSplitter$OutputSink(1258): Waiting for split writer threads to finish 2013-05-24 22:00:39,317 DEBUG [hbase-am-zkevent-worker-pool-2-thread-14] master.AssignmentManager(752): Handling transition=RS_ZK_REGION_OPENING, server=kiyo.gq1.ygridcore.net,42690,1369432806911, region=acf78311e603cb0d02309ae08763d196, current state from region state map ={test,yyy,1369432820321.acf78311e603cb0d02309ae08763d196. state=PENDING_OPEN, ts=1369432835960, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:39,317 INFO [hbase-am-zkevent-worker-pool-2-thread-14] master.RegionStates(264): Region {NAME => 'test,yyy,1369432820321.acf78311e603cb0d02309ae08763d196.', STARTKEY => 'yyy', ENDKEY => 'zzz', ENCODED => acf78311e603cb0d02309ae08763d196,} transitioned from {test,yyy,1369432820321.acf78311e603cb0d02309ae08763d196. state=PENDING_OPEN, ts=1369432835960, server=kiyo.gq1.ygridcore.net,42690,1369432806911} to {test,yyy,1369432820321.acf78311e603cb0d02309ae08763d196. state=OPENING, ts=1369432839317, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:39,328 DEBUG [WriterThread-2-EventThread] zookeeper.ZooKeeperWatcher(307): hconnection-0x43f1a352 Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2013-05-24 22:00:39,329 DEBUG [WriterThread-2-EventThread] zookeeper.ZooKeeperWatcher(384): hconnection-0x43f1a352-0x13ed88e2cdf001c connected 2013-05-24 22:00:39,349 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:42690-0x13ed88e2cdf0006 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/recovering-regions/dba8b67ade91438cad8a5209736330e6 2013-05-24 22:00:39,349 DEBUG [PostOpenDeployTasks:872960a61a4ceeb9f64577d8b7cb0ddf] regionserver.HRegionServer(4268): Update last flushed sequence id of region 872960a61a4ceeb9f64577d8b7cb0ddf for kiyo.gq1.ygridcore.net,49701,1369432806997 2013-05-24 22:00:39,385 ERROR [IPC Server handler 4 on 54096] security.UserGroupInformation(1480): PriviledgedActionException as:hortonzy.hfs.0 (auth:SIMPLE) cause:java.io.FileNotFoundException: File does not exist: /user/hortonzy/hbase/.oldlogs/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432809419 2013-05-24 22:00:39,387 INFO [PostOpenDeployTasks:acf78311e603cb0d02309ae08763d196] regionserver.HRegionServer(1657): Post open deploy tasks for region=test,yyy,1369432820321.acf78311e603cb0d02309ae08763d196. 2013-05-24 22:00:39,387 INFO [ReplicationExecutor-0] replication.ReplicationQueuesZKImpl(152): Moving kiyo.gq1.ygridcore.net,39146,1369432814418's hlogs to my queue 2013-05-24 22:00:39,391 INFO [ReplicationExecutor-0.replicationSource,2-kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.ReplicationSource(501): NB dead servers : 1 2013-05-24 22:00:39,391 INFO [ReplicationExecutor-0.replicationSource,2-kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.ReplicationSource(511): Possible location hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432809419 2013-05-24 22:00:39,393 INFO [ReplicationExecutor-0.replicationSource,2-kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.ReplicationSource(514): Log hdfs://localhost:54096/user/hortonzy/hbase/.oldlogs/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432809419 still exists at hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432809419 2013-05-24 22:00:39,393 DEBUG [ReplicationExecutor-0.replicationSource,2-kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.ReplicationSource(583): Unable to open a reader, sleeping 100 times 1 2013-05-24 22:00:39,395 DEBUG [PostOpenDeployTasks:dba8b67ade91438cad8a5209736330e6] regionserver.HRegionServer(4268): Update last flushed sequence id of region dba8b67ade91438cad8a5209736330e6 for kiyo.gq1.ygridcore.net,49701,1369432806997 2013-05-24 22:00:39,407 INFO [PostOpenDeployTasks:872960a61a4ceeb9f64577d8b7cb0ddf] catalog.MetaEditor(432): Updated row test,sss,1369432820320.872960a61a4ceeb9f64577d8b7cb0ddf. with server=kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:39,407 INFO [PostOpenDeployTasks:dba8b67ade91438cad8a5209736330e6] catalog.MetaEditor(432): Updated row test,uuu,1369432820320.dba8b67ade91438cad8a5209736330e6. with server=kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:39,407 INFO [PostOpenDeployTasks:872960a61a4ceeb9f64577d8b7cb0ddf] regionserver.HRegionServer(1682): Done with post open deploy task for region=test,sss,1369432820320.872960a61a4ceeb9f64577d8b7cb0ddf. 2013-05-24 22:00:39,407 INFO [PostOpenDeployTasks:dba8b67ade91438cad8a5209736330e6] regionserver.HRegionServer(1682): Done with post open deploy task for region=test,uuu,1369432820320.dba8b67ade91438cad8a5209736330e6. 2013-05-24 22:00:39,407 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] zookeeper.ZKAssign(786): regionserver:42690-0x13ed88e2cdf0006 Attempting to transition node 872960a61a4ceeb9f64577d8b7cb0ddf from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-05-24 22:00:39,407 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] zookeeper.ZKAssign(786): regionserver:42690-0x13ed88e2cdf0006 Attempting to transition node dba8b67ade91438cad8a5209736330e6 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-05-24 22:00:39,419 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:42690-0x13ed88e2cdf0006 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/recovering-regions/acf78311e603cb0d02309ae08763d196 2013-05-24 22:00:39,422 DEBUG [WriterThread-2] client.MetaScanner(160): Scanning .META. starting at row=test,,00000000000000 for max=10 with caching=10 2013-05-24 22:00:39,422 DEBUG [WriterThread-2] client.ClientScanner(96): Creating scanner over .META. starting at key 'test,,00000000000000' 2013-05-24 22:00:39,423 DEBUG [WriterThread-2] client.ClientScanner(209): Advancing internal scanner to startKey at 'test,,00000000000000' 2013-05-24 22:00:39,428 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/region-in-transition/872960a61a4ceeb9f64577d8b7cb0ddf 2013-05-24 22:00:39,428 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] zookeeper.ZKAssign(862): regionserver:42690-0x13ed88e2cdf0006 Successfully transitioned node 872960a61a4ceeb9f64577d8b7cb0ddf from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-05-24 22:00:39,428 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] handler.OpenRegionHandler(373): region transitioned to opened in zookeeper: {NAME => 'test,sss,1369432820320.872960a61a4ceeb9f64577d8b7cb0ddf.', STARTKEY => 'sss', ENDKEY => 'ttt', ENCODED => 872960a61a4ceeb9f64577d8b7cb0ddf,}, server: kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:39,428 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] handler.OpenRegionHandler(186): Opened test,sss,1369432820320.872960a61a4ceeb9f64577d8b7cb0ddf. on server:kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:39,428 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] zookeeper.ZKAssign(786): regionserver:42690-0x13ed88e2cdf0006 Attempting to transition node 0b4d70d096cb6cbce3376acd563c77dd from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-05-24 22:00:39,473 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/region-in-transition/dba8b67ade91438cad8a5209736330e6 2013-05-24 22:00:39,473 DEBUG [PostOpenDeployTasks:acf78311e603cb0d02309ae08763d196] regionserver.HRegionServer(4268): Update last flushed sequence id of region acf78311e603cb0d02309ae08763d196 for kiyo.gq1.ygridcore.net,49701,1369432806997 2013-05-24 22:00:39,474 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] zookeeper.ZKAssign(862): regionserver:42690-0x13ed88e2cdf0006 Successfully transitioned node dba8b67ade91438cad8a5209736330e6 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-05-24 22:00:39,474 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] handler.OpenRegionHandler(373): region transitioned to opened in zookeeper: {NAME => 'test,uuu,1369432820320.dba8b67ade91438cad8a5209736330e6.', STARTKEY => 'uuu', ENDKEY => 'vvv', ENCODED => dba8b67ade91438cad8a5209736330e6,}, server: kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:39,475 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] handler.OpenRegionHandler(186): Opened test,uuu,1369432820320.dba8b67ade91438cad8a5209736330e6. on server:kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:39,476 DEBUG [WriterThread-2] client.ClientScanner(196): Finished scanning region {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,} 2013-05-24 22:00:39,476 DEBUG [hbase-am-zkevent-worker-pool-2-thread-19] master.AssignmentManager(752): Handling transition=RS_ZK_REGION_OPENED, server=kiyo.gq1.ygridcore.net,42690,1369432806911, region=872960a61a4ceeb9f64577d8b7cb0ddf, current state from region state map ={test,sss,1369432820320.872960a61a4ceeb9f64577d8b7cb0ddf. state=OPENING, ts=1369432839254, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:39,476 INFO [hbase-am-zkevent-worker-pool-2-thread-19] master.RegionStates(264): Region {NAME => 'test,sss,1369432820320.872960a61a4ceeb9f64577d8b7cb0ddf.', STARTKEY => 'sss', ENDKEY => 'ttt', ENCODED => 872960a61a4ceeb9f64577d8b7cb0ddf,} transitioned from {test,sss,1369432820320.872960a61a4ceeb9f64577d8b7cb0ddf. state=OPENING, ts=1369432839254, server=kiyo.gq1.ygridcore.net,42690,1369432806911} to {test,sss,1369432820320.872960a61a4ceeb9f64577d8b7cb0ddf. state=OPEN, ts=1369432839476, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:39,476 DEBUG [WriterThread-2] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,,1369432825000.45883ac87958f956040ff389c9f7eb86. is kiyo.gq1.ygridcore.net:51523 2013-05-24 22:00:39,476 DEBUG [MASTER_OPEN_REGION-kiyo.gq1.ygridcore.net,49101,1369432806564-1] handler.OpenedRegionHandler(146): Handling OPENED event for test,sss,1369432820320.872960a61a4ceeb9f64577d8b7cb0ddf. from kiyo.gq1.ygridcore.net,42690,1369432806911; deleting unassigned node 2013-05-24 22:00:39,476 DEBUG [MASTER_OPEN_REGION-kiyo.gq1.ygridcore.net,49101,1369432806564-1] zookeeper.ZKAssign(405): master:49101-0x13ed88e2cdf0004 Deleting existing unassigned node for 872960a61a4ceeb9f64577d8b7cb0ddf that is in expected state RS_ZK_REGION_OPENED 2013-05-24 22:00:39,478 DEBUG [hbase-am-zkevent-worker-pool-2-thread-20] master.AssignmentManager(752): Handling transition=RS_ZK_REGION_OPENED, server=kiyo.gq1.ygridcore.net,42690,1369432806911, region=dba8b67ade91438cad8a5209736330e6, current state from region state map ={test,uuu,1369432820320.dba8b67ade91438cad8a5209736330e6. state=OPENING, ts=1369432839255, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:39,478 INFO [hbase-am-zkevent-worker-pool-2-thread-20] master.RegionStates(264): Region {NAME => 'test,uuu,1369432820320.dba8b67ade91438cad8a5209736330e6.', STARTKEY => 'uuu', ENDKEY => 'vvv', ENCODED => dba8b67ade91438cad8a5209736330e6,} transitioned from {test,uuu,1369432820320.dba8b67ade91438cad8a5209736330e6. state=OPENING, ts=1369432839255, server=kiyo.gq1.ygridcore.net,42690,1369432806911} to {test,uuu,1369432820320.dba8b67ade91438cad8a5209736330e6. state=OPEN, ts=1369432839478, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:39,478 DEBUG [MASTER_OPEN_REGION-kiyo.gq1.ygridcore.net,49101,1369432806564-2] handler.OpenedRegionHandler(146): Handling OPENED event for test,uuu,1369432820320.dba8b67ade91438cad8a5209736330e6. from kiyo.gq1.ygridcore.net,42690,1369432806911; deleting unassigned node 2013-05-24 22:00:39,478 DEBUG [MASTER_OPEN_REGION-kiyo.gq1.ygridcore.net,49101,1369432806564-2] zookeeper.ZKAssign(405): master:49101-0x13ed88e2cdf0004 Deleting existing unassigned node for dba8b67ade91438cad8a5209736330e6 that is in expected state RS_ZK_REGION_OPENED 2013-05-24 22:00:39,479 INFO [PostOpenDeployTasks:acf78311e603cb0d02309ae08763d196] catalog.MetaEditor(432): Updated row test,yyy,1369432820321.acf78311e603cb0d02309ae08763d196. with server=kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:39,479 INFO [PostOpenDeployTasks:acf78311e603cb0d02309ae08763d196] regionserver.HRegionServer(1682): Done with post open deploy task for region=test,yyy,1369432820321.acf78311e603cb0d02309ae08763d196. 2013-05-24 22:00:39,479 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] zookeeper.ZKAssign(786): regionserver:42690-0x13ed88e2cdf0006 Attempting to transition node acf78311e603cb0d02309ae08763d196 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-05-24 22:00:39,494 ERROR [IPC Server handler 0 on 54096] security.UserGroupInformation(1480): PriviledgedActionException as:hortonzy.hfs.0 (auth:SIMPLE) cause:java.io.FileNotFoundException: File does not exist: /user/hortonzy/hbase/.oldlogs/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432809419 2013-05-24 22:00:39,495 INFO [ReplicationExecutor-0.replicationSource,2-kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.ReplicationSource(501): NB dead servers : 1 2013-05-24 22:00:39,496 INFO [ReplicationExecutor-0.replicationSource,2-kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.ReplicationSource(511): Possible location hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432809419 2013-05-24 22:00:39,497 INFO [ReplicationExecutor-0.replicationSource,2-kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.ReplicationSource(514): Log hdfs://localhost:54096/user/hortonzy/hbase/.oldlogs/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432809419 still exists at hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432809419 2013-05-24 22:00:39,497 DEBUG [ReplicationExecutor-0.replicationSource,2-kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.ReplicationSource(583): Unable to open a reader, sleeping 100 times 2 2013-05-24 22:00:39,500 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/region-in-transition/0b4d70d096cb6cbce3376acd563c77dd 2013-05-24 22:00:39,501 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] zookeeper.ZKAssign(862): regionserver:42690-0x13ed88e2cdf0006 Successfully transitioned node 0b4d70d096cb6cbce3376acd563c77dd from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-05-24 22:00:39,501 INFO [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegion(4139): HRegion.openHRegion Region name ==test,zzz,1369432820321.0b4d70d096cb6cbce3376acd563c77dd. 2013-05-24 22:00:39,501 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegion(4141): Opening region: {NAME => 'test,zzz,1369432820321.0b4d70d096cb6cbce3376acd563c77dd.', STARTKEY => 'zzz', ENDKEY => '', ENCODED => 0b4d70d096cb6cbce3376acd563c77dd,} 2013-05-24 22:00:39,502 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.MetricsRegionSourceImpl(62): Creating new MetricsRegionSourceImpl for table test 0b4d70d096cb6cbce3376acd563c77dd 2013-05-24 22:00:39,502 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegion(518): Instantiated test,zzz,1369432820321.0b4d70d096cb6cbce3376acd563c77dd. 2013-05-24 22:00:39,506 INFO [StoreOpenerThread-test,zzz,1369432820321.0b4d70d096cb6cbce3376acd563c77dd.-1] compactions.CompactionConfiguration(85): Compaction configuration size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 604800000, major jitter 0.500000 2013-05-24 22:00:39,508 INFO [StoreOpenerThread-test,zzz,1369432820321.0b4d70d096cb6cbce3376acd563c77dd.-1] compactions.CompactionConfiguration(85): Compaction configuration size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 604800000, major jitter 0.500000 2013-05-24 22:00:39,511 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/region-in-transition/872960a61a4ceeb9f64577d8b7cb0ddf 2013-05-24 22:00:39,511 DEBUG [hbase-am-zkevent-worker-pool-2-thread-3] master.AssignmentManager$4(1189): The znode of region test,sss,1369432820320.872960a61a4ceeb9f64577d8b7cb0ddf. has been deleted. 2013-05-24 22:00:39,511 INFO [hbase-am-zkevent-worker-pool-2-thread-3] master.RegionStates(264): Region {NAME => 'test,sss,1369432820320.872960a61a4ceeb9f64577d8b7cb0ddf.', STARTKEY => 'sss', ENDKEY => 'ttt', ENCODED => 872960a61a4ceeb9f64577d8b7cb0ddf,} transitioned from {test,sss,1369432820320.872960a61a4ceeb9f64577d8b7cb0ddf. state=OPEN, ts=1369432839476, server=kiyo.gq1.ygridcore.net,42690,1369432806911} to {test,sss,1369432820320.872960a61a4ceeb9f64577d8b7cb0ddf. state=OPEN, ts=1369432839511, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:39,511 INFO [hbase-am-zkevent-worker-pool-2-thread-3] master.RegionStates(299): Onlined region {NAME => 'test,sss,1369432820320.872960a61a4ceeb9f64577d8b7cb0ddf.', STARTKEY => 'sss', ENDKEY => 'ttt', ENCODED => 872960a61a4ceeb9f64577d8b7cb0ddf,} on kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:39,511 INFO [hbase-am-zkevent-worker-pool-2-thread-3] master.AssignmentManager$4(1194): The master has opened the region test,sss,1369432820320.872960a61a4ceeb9f64577d8b7cb0ddf. that was online on kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:39,511 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.AssignmentManager(2753): Waiting on null to clear regions-in-transition 2013-05-24 22:00:39,511 INFO [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegion(608): Onlined test,zzz,1369432820321.0b4d70d096cb6cbce3376acd563c77dd.; next sequenceid=1 2013-05-24 22:00:39,512 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] zookeeper.ZKAssign(598): regionserver:42690-0x13ed88e2cdf0006 Attempting to retransition the opening state of node 0b4d70d096cb6cbce3376acd563c77dd 2013-05-24 22:00:39,512 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/region-in-transition 2013-05-24 22:00:39,512 DEBUG [MASTER_OPEN_REGION-kiyo.gq1.ygridcore.net,49101,1369432806564-1] zookeeper.ZKAssign(434): master:49101-0x13ed88e2cdf0004 Successfully deleted unassigned node for region 872960a61a4ceeb9f64577d8b7cb0ddf in expected state RS_ZK_REGION_OPENED 2013-05-24 22:00:39,513 DEBUG [hbase-am-zkevent-worker-pool-2-thread-7] master.AssignmentManager(752): Handling transition=RS_ZK_REGION_OPENING, server=kiyo.gq1.ygridcore.net,42690,1369432806911, region=0b4d70d096cb6cbce3376acd563c77dd, current state from region state map ={test,zzz,1369432820321.0b4d70d096cb6cbce3376acd563c77dd. state=PENDING_OPEN, ts=1369432835960, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:39,513 INFO [hbase-am-zkevent-worker-pool-2-thread-7] master.RegionStates(264): Region {NAME => 'test,zzz,1369432820321.0b4d70d096cb6cbce3376acd563c77dd.', STARTKEY => 'zzz', ENDKEY => '', ENCODED => 0b4d70d096cb6cbce3376acd563c77dd,} transitioned from {test,zzz,1369432820321.0b4d70d096cb6cbce3376acd563c77dd. state=PENDING_OPEN, ts=1369432835960, server=kiyo.gq1.ygridcore.net,42690,1369432806911} to {test,zzz,1369432820321.0b4d70d096cb6cbce3376acd563c77dd. state=OPENING, ts=1369432839513, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:39,519 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/region-in-transition/dba8b67ade91438cad8a5209736330e6 2013-05-24 22:00:39,519 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:51523-0x13ed88e2cdf0010 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/2/replication/rs/kiyo.gq1.ygridcore.net,39146,1369432814418/lock 2013-05-24 22:00:39,519 DEBUG [hbase-am-zkevent-worker-pool-2-thread-5] master.AssignmentManager$4(1189): The znode of region test,uuu,1369432820320.dba8b67ade91438cad8a5209736330e6. has been deleted. 2013-05-24 22:00:39,520 INFO [hbase-am-zkevent-worker-pool-2-thread-5] master.RegionStates(264): Region {NAME => 'test,uuu,1369432820320.dba8b67ade91438cad8a5209736330e6.', STARTKEY => 'uuu', ENDKEY => 'vvv', ENCODED => dba8b67ade91438cad8a5209736330e6,} transitioned from {test,uuu,1369432820320.dba8b67ade91438cad8a5209736330e6. state=OPEN, ts=1369432839478, server=kiyo.gq1.ygridcore.net,42690,1369432806911} to {test,uuu,1369432820320.dba8b67ade91438cad8a5209736330e6. state=OPEN, ts=1369432839520, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:39,520 INFO [hbase-am-zkevent-worker-pool-2-thread-5] master.RegionStates(299): Onlined region {NAME => 'test,uuu,1369432820320.dba8b67ade91438cad8a5209736330e6.', STARTKEY => 'uuu', ENDKEY => 'vvv', ENCODED => dba8b67ade91438cad8a5209736330e6,} on kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:39,520 DEBUG [MASTER_OPEN_REGION-kiyo.gq1.ygridcore.net,49101,1369432806564-2] zookeeper.ZKAssign(434): master:49101-0x13ed88e2cdf0004 Successfully deleted unassigned node for region dba8b67ade91438cad8a5209736330e6 in expected state RS_ZK_REGION_OPENED 2013-05-24 22:00:39,520 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.AssignmentManager(2753): Waiting on null to clear regions-in-transition 2013-05-24 22:00:39,520 INFO [hbase-am-zkevent-worker-pool-2-thread-5] master.AssignmentManager$4(1194): The master has opened the region test,uuu,1369432820320.dba8b67ade91438cad8a5209736330e6. that was online on kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:39,527 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/region-in-transition/acf78311e603cb0d02309ae08763d196 2013-05-24 22:00:39,528 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] zookeeper.ZKAssign(862): regionserver:42690-0x13ed88e2cdf0006 Successfully transitioned node acf78311e603cb0d02309ae08763d196 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-05-24 22:00:39,528 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] handler.OpenRegionHandler(373): region transitioned to opened in zookeeper: {NAME => 'test,yyy,1369432820321.acf78311e603cb0d02309ae08763d196.', STARTKEY => 'yyy', ENDKEY => 'zzz', ENCODED => acf78311e603cb0d02309ae08763d196,}, server: kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:39,528 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] handler.OpenRegionHandler(186): Opened test,yyy,1369432820321.acf78311e603cb0d02309ae08763d196. on server:kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:39,529 INFO [PostOpenDeployTasks:0b4d70d096cb6cbce3376acd563c77dd] regionserver.HRegionServer(1657): Post open deploy tasks for region=test,zzz,1369432820321.0b4d70d096cb6cbce3376acd563c77dd. 2013-05-24 22:00:39,537 DEBUG [hbase-am-zkevent-worker-pool-2-thread-9] master.AssignmentManager(752): Handling transition=RS_ZK_REGION_OPENED, server=kiyo.gq1.ygridcore.net,42690,1369432806911, region=acf78311e603cb0d02309ae08763d196, current state from region state map ={test,yyy,1369432820321.acf78311e603cb0d02309ae08763d196. state=OPENING, ts=1369432839317, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:39,537 INFO [hbase-am-zkevent-worker-pool-2-thread-9] master.RegionStates(264): Region {NAME => 'test,yyy,1369432820321.acf78311e603cb0d02309ae08763d196.', STARTKEY => 'yyy', ENDKEY => 'zzz', ENCODED => acf78311e603cb0d02309ae08763d196,} transitioned from {test,yyy,1369432820321.acf78311e603cb0d02309ae08763d196. state=OPENING, ts=1369432839317, server=kiyo.gq1.ygridcore.net,42690,1369432806911} to {test,yyy,1369432820321.acf78311e603cb0d02309ae08763d196. state=OPEN, ts=1369432839537, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:39,537 DEBUG [MASTER_OPEN_REGION-kiyo.gq1.ygridcore.net,49101,1369432806564-3] handler.OpenedRegionHandler(146): Handling OPENED event for test,yyy,1369432820321.acf78311e603cb0d02309ae08763d196. from kiyo.gq1.ygridcore.net,42690,1369432806911; deleting unassigned node 2013-05-24 22:00:39,537 DEBUG [MASTER_OPEN_REGION-kiyo.gq1.ygridcore.net,49101,1369432806564-3] zookeeper.ZKAssign(405): master:49101-0x13ed88e2cdf0004 Deleting existing unassigned node for acf78311e603cb0d02309ae08763d196 that is in expected state RS_ZK_REGION_OPENED 2013-05-24 22:00:39,550 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:42690-0x13ed88e2cdf0006 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/recovering-regions/0b4d70d096cb6cbce3376acd563c77dd 2013-05-24 22:00:39,561 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/region-in-transition/acf78311e603cb0d02309ae08763d196 2013-05-24 22:00:39,561 DEBUG [hbase-am-zkevent-worker-pool-2-thread-4] master.AssignmentManager$4(1189): The znode of region test,yyy,1369432820321.acf78311e603cb0d02309ae08763d196. has been deleted. 2013-05-24 22:00:39,561 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/region-in-transition 2013-05-24 22:00:39,561 INFO [hbase-am-zkevent-worker-pool-2-thread-4] master.RegionStates(264): Region {NAME => 'test,yyy,1369432820321.acf78311e603cb0d02309ae08763d196.', STARTKEY => 'yyy', ENDKEY => 'zzz', ENCODED => acf78311e603cb0d02309ae08763d196,} transitioned from {test,yyy,1369432820321.acf78311e603cb0d02309ae08763d196. state=OPEN, ts=1369432839537, server=kiyo.gq1.ygridcore.net,42690,1369432806911} to {test,yyy,1369432820321.acf78311e603cb0d02309ae08763d196. state=OPEN, ts=1369432839561, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:39,562 INFO [hbase-am-zkevent-worker-pool-2-thread-4] master.RegionStates(299): Onlined region {NAME => 'test,yyy,1369432820321.acf78311e603cb0d02309ae08763d196.', STARTKEY => 'yyy', ENDKEY => 'zzz', ENCODED => acf78311e603cb0d02309ae08763d196,} on kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:39,562 INFO [hbase-am-zkevent-worker-pool-2-thread-4] master.AssignmentManager$4(1194): The master has opened the region test,yyy,1369432820321.acf78311e603cb0d02309ae08763d196. that was online on kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:39,562 DEBUG [MASTER_OPEN_REGION-kiyo.gq1.ygridcore.net,49101,1369432806564-3] zookeeper.ZKAssign(434): master:49101-0x13ed88e2cdf0004 Successfully deleted unassigned node for region acf78311e603cb0d02309ae08763d196 in expected state RS_ZK_REGION_OPENED 2013-05-24 22:00:39,562 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.AssignmentManager(2753): Waiting on null to clear regions-in-transition 2013-05-24 22:00:39,569 DEBUG [PostOpenDeployTasks:0b4d70d096cb6cbce3376acd563c77dd] regionserver.HRegionServer(4268): Update last flushed sequence id of region 0b4d70d096cb6cbce3376acd563c77dd for kiyo.gq1.ygridcore.net,49701,1369432806997 2013-05-24 22:00:39,573 INFO [PostOpenDeployTasks:0b4d70d096cb6cbce3376acd563c77dd] catalog.MetaEditor(432): Updated row test,zzz,1369432820321.0b4d70d096cb6cbce3376acd563c77dd. with server=kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:39,573 INFO [PostOpenDeployTasks:0b4d70d096cb6cbce3376acd563c77dd] regionserver.HRegionServer(1682): Done with post open deploy task for region=test,zzz,1369432820321.0b4d70d096cb6cbce3376acd563c77dd. 2013-05-24 22:00:39,573 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] zookeeper.ZKAssign(786): regionserver:42690-0x13ed88e2cdf0006 Attempting to transition node 0b4d70d096cb6cbce3376acd563c77dd from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-05-24 22:00:39,588 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/region-in-transition/0b4d70d096cb6cbce3376acd563c77dd 2013-05-24 22:00:39,589 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] zookeeper.ZKAssign(862): regionserver:42690-0x13ed88e2cdf0006 Successfully transitioned node 0b4d70d096cb6cbce3376acd563c77dd from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-05-24 22:00:39,589 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] handler.OpenRegionHandler(373): region transitioned to opened in zookeeper: {NAME => 'test,zzz,1369432820321.0b4d70d096cb6cbce3376acd563c77dd.', STARTKEY => 'zzz', ENDKEY => '', ENCODED => 0b4d70d096cb6cbce3376acd563c77dd,}, server: kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:39,589 DEBUG [RS_OPEN_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] handler.OpenRegionHandler(186): Opened test,zzz,1369432820321.0b4d70d096cb6cbce3376acd563c77dd. on server:kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:39,590 DEBUG [hbase-am-zkevent-worker-pool-2-thread-12] master.AssignmentManager(752): Handling transition=RS_ZK_REGION_OPENED, server=kiyo.gq1.ygridcore.net,42690,1369432806911, region=0b4d70d096cb6cbce3376acd563c77dd, current state from region state map ={test,zzz,1369432820321.0b4d70d096cb6cbce3376acd563c77dd. state=OPENING, ts=1369432839513, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:39,591 INFO [hbase-am-zkevent-worker-pool-2-thread-12] master.RegionStates(264): Region {NAME => 'test,zzz,1369432820321.0b4d70d096cb6cbce3376acd563c77dd.', STARTKEY => 'zzz', ENDKEY => '', ENCODED => 0b4d70d096cb6cbce3376acd563c77dd,} transitioned from {test,zzz,1369432820321.0b4d70d096cb6cbce3376acd563c77dd. state=OPENING, ts=1369432839513, server=kiyo.gq1.ygridcore.net,42690,1369432806911} to {test,zzz,1369432820321.0b4d70d096cb6cbce3376acd563c77dd. state=OPEN, ts=1369432839591, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:39,591 DEBUG [MASTER_OPEN_REGION-kiyo.gq1.ygridcore.net,49101,1369432806564-4] handler.OpenedRegionHandler(146): Handling OPENED event for test,zzz,1369432820321.0b4d70d096cb6cbce3376acd563c77dd. from kiyo.gq1.ygridcore.net,42690,1369432806911; deleting unassigned node 2013-05-24 22:00:39,591 DEBUG [MASTER_OPEN_REGION-kiyo.gq1.ygridcore.net,49101,1369432806564-4] zookeeper.ZKAssign(405): master:49101-0x13ed88e2cdf0004 Deleting existing unassigned node for 0b4d70d096cb6cbce3376acd563c77dd that is in expected state RS_ZK_REGION_OPENED 2013-05-24 22:00:39,608 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/region-in-transition/0b4d70d096cb6cbce3376acd563c77dd 2013-05-24 22:00:39,609 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/region-in-transition 2013-05-24 22:00:39,609 DEBUG [hbase-am-zkevent-worker-pool-2-thread-10] master.AssignmentManager$4(1189): The znode of region test,zzz,1369432820321.0b4d70d096cb6cbce3376acd563c77dd. has been deleted. 2013-05-24 22:00:39,609 DEBUG [MASTER_OPEN_REGION-kiyo.gq1.ygridcore.net,49101,1369432806564-4] zookeeper.ZKAssign(434): master:49101-0x13ed88e2cdf0004 Successfully deleted unassigned node for region 0b4d70d096cb6cbce3376acd563c77dd in expected state RS_ZK_REGION_OPENED 2013-05-24 22:00:39,609 INFO [hbase-am-zkevent-worker-pool-2-thread-10] master.RegionStates(264): Region {NAME => 'test,zzz,1369432820321.0b4d70d096cb6cbce3376acd563c77dd.', STARTKEY => 'zzz', ENDKEY => '', ENCODED => 0b4d70d096cb6cbce3376acd563c77dd,} transitioned from {test,zzz,1369432820321.0b4d70d096cb6cbce3376acd563c77dd. state=OPEN, ts=1369432839591, server=kiyo.gq1.ygridcore.net,42690,1369432806911} to {test,zzz,1369432820321.0b4d70d096cb6cbce3376acd563c77dd. state=OPEN, ts=1369432839609, server=kiyo.gq1.ygridcore.net,42690,1369432806911} 2013-05-24 22:00:39,609 INFO [hbase-am-zkevent-worker-pool-2-thread-10] master.RegionStates(299): Onlined region {NAME => 'test,zzz,1369432820321.0b4d70d096cb6cbce3376acd563c77dd.', STARTKEY => 'zzz', ENDKEY => '', ENCODED => 0b4d70d096cb6cbce3376acd563c77dd,} on kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:39,610 INFO [hbase-am-zkevent-worker-pool-2-thread-10] master.AssignmentManager$4(1194): The master has opened the region test,zzz,1369432820321.0b4d70d096cb6cbce3376acd563c77dd. that was online on kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:39,635 DEBUG [kiyo.gq1.ygridcore.net,53632,1369432814258.splitLogManagerTimeoutMonitor] master.SplitLogManager$TimeoutMonitor(1337): total tasks = 6 unassigned = 5 2013-05-24 22:00:39,645 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.MasterFileSystem(322): Renamed region directory: hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997-splitting 2013-05-24 22:00:39,646 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.SplitLogManager(1283): dead splitlog workers [kiyo.gq1.ygridcore.net,49701,1369432806997] 2013-05-24 22:00:39,646 DEBUG [WriterThread-2] wal.WALEditsReplaySink(128): number of rows:1023 are sent by batch! spent 120(ms)! 2013-05-24 22:00:39,648 DEBUG [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.SplitLogManager(304): Scheduling batch of logs to split 2013-05-24 22:00:39,648 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.SplitLogManager(306): started splitting 4 logs in [hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997-splitting] 2013-05-24 22:00:39,658 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:42690-0x13ed88e2cdf0006 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/splitlog 2013-05-24 22:00:39,659 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] regionserver.SplitLogWorker(572): tasks arrived or departed 2013-05-24 22:00:39,659 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(666): put up splitlog task at znode /1/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C49701%2C1369432806997-splitting%2Fkiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432809419 2013-05-24 22:00:39,675 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(666): put up splitlog task at znode /1/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C49701%2C1369432806997-splitting%2Fkiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432829856 2013-05-24 22:00:39,676 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(666): put up splitlog task at znode /1/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C49701%2C1369432806997-splitting%2Fkiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432832342 2013-05-24 22:00:39,677 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(666): put up splitlog task at znode /1/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C49701%2C1369432806997-splitting%2Fkiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432833549 2013-05-24 22:00:39,679 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(708): task not yet acquired /1/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C49701%2C1369432806997-splitting%2Fkiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432809419 ver = 0 2013-05-24 22:00:39,679 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(708): task not yet acquired /1/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C49701%2C1369432806997-splitting%2Fkiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432829856 ver = 0 2013-05-24 22:00:39,679 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(708): task not yet acquired /1/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C49701%2C1369432806997-splitting%2Fkiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432832342 ver = 0 2013-05-24 22:00:39,680 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(708): task not yet acquired /1/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C49701%2C1369432806997-splitting%2Fkiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432833549 ver = 0 2013-05-24 22:00:39,692 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C49701%2C1369432806997-splitting%2Fkiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432829856 2013-05-24 22:00:39,692 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,42690,1369432806911] regionserver.SplitLogWorker(330): worker kiyo.gq1.ygridcore.net,42690,1369432806911 acquired task /1/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C49701%2C1369432806997-splitting%2Fkiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432829856 2013-05-24 22:00:39,693 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(781): task /1/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C49701%2C1369432806997-splitting%2Fkiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432829856 acquired by kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:39,694 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,42690,1369432806911] wal.HLogSplitter(506): Splitting hlog: hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997-splitting/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432829856, length=46234 2013-05-24 22:00:39,694 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,42690,1369432806911] wal.HLogSplitter(507): DistributedLogReplay = true 2013-05-24 22:00:39,698 ERROR [IPC Server handler 2 on 54096] security.UserGroupInformation(1480): PriviledgedActionException as:hortonzy.hfs.0 (auth:SIMPLE) cause:java.io.FileNotFoundException: File does not exist: /user/hortonzy/hbase/.oldlogs/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432809419 2013-05-24 22:00:39,701 INFO [ReplicationExecutor-0.replicationSource,2-kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.ReplicationSource(501): NB dead servers : 1 2013-05-24 22:00:39,701 INFO [ReplicationExecutor-0.replicationSource,2-kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.ReplicationSource(511): Possible location hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432809419 2013-05-24 22:00:39,702 INFO [ReplicationExecutor-0.replicationSource,2-kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.ReplicationSource(511): Possible location hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997-splitting/kiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432809419 2013-05-24 22:00:39,703 INFO [ReplicationExecutor-0.replicationSource,2-kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.ReplicationSource(514): Log hdfs://localhost:54096/user/hortonzy/hbase/.oldlogs/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432809419 still exists at hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997-splitting/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432809419 2013-05-24 22:00:39,703 DEBUG [ReplicationExecutor-0.replicationSource,2-kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.ReplicationSource(583): Unable to open a reader, sleeping 100 times 3 2013-05-24 22:00:39,708 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:42690-0x13ed88e2cdf0006 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C49701%2C1369432806997-splitting%2Fkiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432829856 2013-05-24 22:00:39,708 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C49701%2C1369432806997-splitting%2Fkiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432829856 2013-05-24 22:00:39,709 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,42690,1369432806911] util.FSHDFSUtils(86): Recovering lease on dfs file hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997-splitting/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432829856 2013-05-24 22:00:39,710 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,42690,1369432806911] util.FSHDFSUtils(156): recoverLease=true, attempt=0 on file=hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997-splitting/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432829856 after 0ms 2013-05-24 22:00:39,717 DEBUG [SplitLogWorker-kiyo.gq1.ygridcore.net,42690,1369432806911] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 46226, fileLength: 46234, trailerPresent: true 2013-05-24 22:00:39,718 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392.logRoller] regionserver.LogRoller(90): HLog roll requested 2013-05-24 22:00:39,718 DEBUG [WriterThread-2] wal.WALEditsReplaySink(128): number of rows:864 are sent by batch! spent 60(ms)! 2013-05-24 22:00:39,719 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] wal.HLogSplitter$OutputSink(1276): Split writers finished 2013-05-24 22:00:39,720 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] client.HConnectionManager$HConnectionImplementation(1640): Closing zookeeper sessionid=0x13ed88e2cdf001c 2013-05-24 22:00:39,736 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] wal.HLogSplitter(602): Processed 8 edits across 1 regions; log file=hdfs://localhost:54104/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,39146,1369432814418-splitting/kiyo.gq1.ygridcore.net%2C39146%2C1369432814418.1369432829746 is corrupted = false progress failed = false 2013-05-24 22:00:39,738 DEBUG [WriterThread-0] wal.HLogSplitter$WriterThread(1102): Writer thread Thread[WriterThread-0,5,main]: starting 2013-05-24 22:00:39,738 DEBUG [WriterThread-1] wal.HLogSplitter$WriterThread(1102): Writer thread Thread[WriterThread-1,5,main]: starting 2013-05-24 22:00:39,738 DEBUG [WriterThread-2] wal.HLogSplitter$WriterThread(1102): Writer thread Thread[WriterThread-2,5,main]: starting 2013-05-24 22:00:39,756 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:51523-0x13ed88e2cdf0010 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432829746 2013-05-24 22:00:39,756 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:53632-0x13ed88e2cdf000f Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432829746 2013-05-24 22:00:39,757 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] regionserver.SplitLogWorker(451): successfully transitioned task /2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432829746 to final state DONE kiyo.gq1.ygridcore.net,51523,1369432814392 2013-05-24 22:00:39,757 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(716): task /2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432829746 entered state: DONE kiyo.gq1.ygridcore.net,51523,1369432814392 2013-05-24 22:00:39,757 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] regionserver.SplitLogWorker(385): worker kiyo.gq1.ygridcore.net,51523,1369432814392 done with task /2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432829746 in 504ms 2013-05-24 22:00:39,758 INFO [WriterThread-2] zookeeper.RecoverableZooKeeper(119): Process identifier=hconnection-0x321f8d38 connecting to ZooKeeper ensemble=localhost:51411 2013-05-24 22:00:39,766 INFO [IPC Server handler 6 on 54104] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52910 is added to blk_-8498118074637429680_1015{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:33054|RBW], ReplicaUnderConstruction[127.0.0.1:52910|RBW]]} size 0 2013-05-24 22:00:39,767 INFO [IPC Server handler 0 on 54104] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33054 is added to blk_-8498118074637429680_1015 size 43543 2013-05-24 22:00:39,779 DEBUG [WriterThread-2-EventThread] zookeeper.ZooKeeperWatcher(307): hconnection-0x321f8d38 Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2013-05-24 22:00:39,780 DEBUG [WriterThread-2-EventThread] zookeeper.ZooKeeperWatcher(384): hconnection-0x321f8d38-0x13ed88e2cdf001d connected 2013-05-24 22:00:39,791 INFO [kiyo.gq1.ygridcore.net,42690,1369432806911Replication Statistics #0] regionserver.Replication$ReplicationStatisticsThread(291): Normal source for cluster 2: Total replicated edits: 2115, currently replicating from: hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432834215 at position: N/A Recovered source for cluster/machine(s) 2: Total replicated edits: 0, currently replicating from: hdfs://localhost:54096/user/hortonzy/hbase/.oldlogs/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432809419 at position: N/A 2013-05-24 22:00:39,791 INFO [kiyo.gq1.ygridcore.net,49701,1369432806997Replication Statistics #0] regionserver.Replication$ReplicationStatisticsThread(291): Normal source for cluster 2: Total replicated edits: 1629, currently replicating from: hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432832342 at position: N/A 2013-05-24 22:00:39,848 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] regionserver.SplitLogWorker(330): worker kiyo.gq1.ygridcore.net,51523,1369432814392 acquired task /2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432831597 2013-05-24 22:00:39,849 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] wal.HLogSplitter(506): Splitting hlog: hdfs://localhost:54104/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,39146,1369432814418-splitting/kiyo.gq1.ygridcore.net%2C39146%2C1369432814418.1369432831597, length=42319 2013-05-24 22:00:39,849 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] wal.HLogSplitter(507): DistributedLogReplay = true 2013-05-24 22:00:39,853 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,42690,1369432806911] wal.HLogSplitter(595): Finishing writing output logs and closing down. 2013-05-24 22:00:39,854 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,42690,1369432806911] wal.HLogSplitter$OutputSink(1258): Waiting for split writer threads to finish 2013-05-24 22:00:39,891 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392.logRoller] wal.FSHLog(522): Rolled WAL /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,51523,1369432814392/kiyo.gq1.ygridcore.net%2C51523%2C1369432814392.1369432818222, entries=2, filesize=42.5 K; new WAL=/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,51523,1369432814392/kiyo.gq1.ygridcore.net%2C51523%2C1369432814392.1369432839718 2013-05-24 22:00:39,902 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:51523-0x13ed88e2cdf0010 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432831597 2013-05-24 22:00:39,903 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] util.FSHDFSUtils(86): Recovering lease on dfs file hdfs://localhost:54104/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,39146,1369432814418-splitting/kiyo.gq1.ygridcore.net%2C39146%2C1369432814418.1369432831597 2013-05-24 22:00:39,904 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] util.FSHDFSUtils(156): recoverLease=true, attempt=0 on file=hdfs://localhost:54104/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,39146,1369432814418-splitting/kiyo.gq1.ygridcore.net%2C39146%2C1369432814418.1369432831597 after 1ms 2013-05-24 22:00:39,906 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432834215 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:39,907 DEBUG [WriterThread-2] client.MetaScanner(160): Scanning .META. starting at row=test,ggg,00000000000000 for max=10 with caching=10 2013-05-24 22:00:39,907 DEBUG [WriterThread-2] client.ClientScanner(96): Creating scanner over .META. starting at key 'test,ggg,00000000000000' 2013-05-24 22:00:39,907 DEBUG [WriterThread-2] client.ClientScanner(209): Advancing internal scanner to startKey at 'test,ggg,00000000000000' 2013-05-24 22:00:39,908 DEBUG [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 42311, fileLength: 42319, trailerPresent: true 2013-05-24 22:00:39,909 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 0, fileLength: 0, trailerPresent: false 2013-05-24 22:00:39,913 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(583): Nothing to replicate, sleeping 100 times 10 2013-05-24 22:00:39,913 DEBUG [WriterThread-2] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,ggg,1369432820319.4b3ff3efde12c5afe1220e81d155ac73. is kiyo.gq1.ygridcore.net:42690 2013-05-24 22:00:39,913 DEBUG [WriterThread-2] wal.HLogSplitter$WriterThread(1102): Writer thread Thread[WriterThread-2,5,main]: starting 2013-05-24 22:00:39,913 DEBUG [WriterThread-1] wal.HLogSplitter$WriterThread(1102): Writer thread Thread[WriterThread-1,5,main]: starting 2013-05-24 22:00:39,913 DEBUG [WriterThread-0] wal.HLogSplitter$WriterThread(1102): Writer thread Thread[WriterThread-0,5,main]: starting 2013-05-24 22:00:39,913 DEBUG [WriterThread-2] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,hhh,1369432820320.a9a8fa0d1fb6a035656bc264ec313d2b. is kiyo.gq1.ygridcore.net:42690 2013-05-24 22:00:39,915 DEBUG [WriterThread-2] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,iii,1369432820320.992fe9e8d6a399ebc508d777668f74ad. is kiyo.gq1.ygridcore.net:42690 2013-05-24 22:00:39,915 DEBUG [WriterThread-2] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,jjj,1369432820320.eb38c552d2a930aa1ce862202da80908. is kiyo.gq1.ygridcore.net:42690 2013-05-24 22:00:39,916 DEBUG [WriterThread-2] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,kkk,1369432820320.1ce9aafaaa422cef4ce0e1895111c733. is kiyo.gq1.ygridcore.net:42690 2013-05-24 22:00:39,916 DEBUG [WriterThread-2] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,lll,1369432820320.307a0833b0558971bccf01de5c996ebc. is kiyo.gq1.ygridcore.net:42690 2013-05-24 22:00:39,916 DEBUG [WriterThread-2] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,mmm,1369432820320.ab7b9b9773243cff0ac09f3c379e33dc. is kiyo.gq1.ygridcore.net:42690 2013-05-24 22:00:39,917 DEBUG [WriterThread-2] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,nnn,1369432820320.53e3aef4cc8815dd31923659bff5ca21. is kiyo.gq1.ygridcore.net:42690 2013-05-24 22:00:39,917 DEBUG [WriterThread-2] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,ooo,1369432820320.0d6df81a73ad3b2a79add80716da62ac. is kiyo.gq1.ygridcore.net:42690 2013-05-24 22:00:39,917 DEBUG [WriterThread-2] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,ppp,1369432820320.da89687648e34409286fd02380093aed. is kiyo.gq1.ygridcore.net:42690 2013-05-24 22:00:39,918 INFO [WriterThread-0] zookeeper.RecoverableZooKeeper(119): Process identifier=hconnection-0x22e2d428 connecting to ZooKeeper ensemble=localhost:51411 2013-05-24 22:00:39,921 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] wal.HLogSplitter(595): Finishing writing output logs and closing down. 2013-05-24 22:00:39,921 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] wal.HLogSplitter$OutputSink(1258): Waiting for split writer threads to finish 2013-05-24 22:00:39,933 DEBUG [WriterThread-0-EventThread] zookeeper.ZooKeeperWatcher(307): hconnection-0x22e2d428 Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2013-05-24 22:00:39,935 DEBUG [WriterThread-0-EventThread] zookeeper.ZooKeeperWatcher(384): hconnection-0x22e2d428-0x13ed88e2cdf001e connected 2013-05-24 22:00:39,945 DEBUG [WriterThread-0] client.MetaScanner(160): Scanning .META. starting at row=test,,00000000000000 for max=10 with caching=10 2013-05-24 22:00:39,945 DEBUG [WriterThread-0] client.ClientScanner(96): Creating scanner over .META. starting at key 'test,,00000000000000' 2013-05-24 22:00:39,945 DEBUG [WriterThread-0] client.ClientScanner(209): Advancing internal scanner to startKey at 'test,,00000000000000' 2013-05-24 22:00:39,950 DEBUG [WriterThread-0] client.ClientScanner(196): Finished scanning region {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,} 2013-05-24 22:00:39,951 DEBUG [WriterThread-0] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,,1369432825000.45883ac87958f956040ff389c9f7eb86. is kiyo.gq1.ygridcore.net:51523 2013-05-24 22:00:39,951 DEBUG [pool-1-thread-1-EventThread] wal.HLogSplitter(692): Archived processed log hdfs://localhost:54104/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,39146,1369432814418-splitting/kiyo.gq1.ygridcore.net%2C39146%2C1369432814418.1369432829746 to hdfs://localhost:54104/user/hortonzy/hbase/.oldlogs/kiyo.gq1.ygridcore.net%2C39146%2C1369432814418.1369432829746 2013-05-24 22:00:39,953 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(629): Done splitting /2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432829746 2013-05-24 22:00:39,954 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:53632-0x13ed88e2cdf000f Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432831597 2013-05-24 22:00:39,958 DEBUG [WriterThread-2] wal.WALEditsReplaySink(128): number of rows:175 are sent by batch! spent 17(ms)! 2013-05-24 22:00:39,972 DEBUG [WriterThread-0] wal.WALEditsReplaySink(128): number of rows:146 are sent by batch! spent 14(ms)! 2013-05-24 22:00:39,975 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:53632-0x13ed88e2cdf000f Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432829746 2013-05-24 22:00:39,975 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:51523-0x13ed88e2cdf0010 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/2/splitlog 2013-05-24 22:00:39,975 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392-EventThread] regionserver.SplitLogWorker(572): tasks arrived or departed 2013-05-24 22:00:39,975 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager$DeleteAsyncCallback(1529): deleted /2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432829746 2013-05-24 22:00:39,976 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(781): task /2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432831597 acquired by kiyo.gq1.ygridcore.net,51523,1369432814392 2013-05-24 22:00:39,980 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.logRoller] regionserver.LogRoller(90): HLog roll requested 2013-05-24 22:00:40,004 ERROR [IPC Server handler 6 on 54096] security.UserGroupInformation(1480): PriviledgedActionException as:hortonzy.hfs.0 (auth:SIMPLE) cause:java.io.FileNotFoundException: File does not exist: /user/hortonzy/hbase/.oldlogs/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432809419 2013-05-24 22:00:40,007 INFO [ReplicationExecutor-0.replicationSource,2-kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.ReplicationSource(501): NB dead servers : 1 2013-05-24 22:00:40,008 INFO [ReplicationExecutor-0.replicationSource,2-kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.ReplicationSource(511): Possible location hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432809419 2013-05-24 22:00:40,036 INFO [ReplicationExecutor-0.replicationSource,2-kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.ReplicationSource(511): Possible location hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997-splitting/kiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432809419 2013-05-24 22:00:40,039 INFO [ReplicationExecutor-0.replicationSource,2-kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.ReplicationSource(514): Log hdfs://localhost:54096/user/hortonzy/hbase/.oldlogs/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432809419 still exists at hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997-splitting/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432809419 2013-05-24 22:00:40,040 DEBUG [ReplicationExecutor-0.replicationSource,2-kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.ReplicationSource(583): Unable to open a reader, sleeping 100 times 4 2013-05-24 22:00:40,040 INFO [IPC Server handler 5 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59993 is added to blk_5383739380851076376_1042{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:53333|RBW], ReplicaUnderConstruction[127.0.0.1:59993|RBW]]} size 0 2013-05-24 22:00:40,040 INFO [IPC Server handler 9 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53333 is added to blk_5383739380851076376_1042 size 44130 2013-05-24 22:00:40,061 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.logRoller] wal.FSHLog(522): Rolled WAL /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432834215, entries=240, filesize=43.1 K; new WAL=/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432839980 2013-05-24 22:00:40,072 DEBUG [kiyo.gq1.ygridcore.net,49101,1369432806564.splitLogManagerTimeoutMonitor] master.SplitLogManager$TimeoutMonitor(1337): total tasks = 4 unassigned = 3 2013-05-24 22:00:40,072 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392.logRoller] regionserver.LogRoller(90): HLog roll requested 2013-05-24 22:00:40,073 DEBUG [WriterThread-0] wal.WALEditsReplaySink(128): number of rows:1824 are sent by batch! spent 101(ms)! 2013-05-24 22:00:40,073 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] wal.HLogSplitter$OutputSink(1276): Split writers finished 2013-05-24 22:00:40,074 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] client.HConnectionManager$HConnectionImplementation(1640): Closing zookeeper sessionid=0x13ed88e2cdf001e 2013-05-24 22:00:40,100 DEBUG [WriterThread-1] wal.WALEditsReplaySink(128): number of rows:1089 are sent by batch! spent 148(ms)! 2013-05-24 22:00:40,101 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,42690,1369432806911] wal.HLogSplitter$OutputSink(1276): Split writers finished 2013-05-24 22:00:40,101 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,42690,1369432806911] client.HConnectionManager$HConnectionImplementation(1640): Closing zookeeper sessionid=0x13ed88e2cdf001d 2013-05-24 22:00:40,107 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] wal.HLogSplitter(602): Processed 6 edits across 1 regions; log file=hdfs://localhost:54104/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,39146,1369432814418-splitting/kiyo.gq1.ygridcore.net%2C39146%2C1369432814418.1369432831597 is corrupted = false progress failed = false 2013-05-24 22:00:40,110 INFO [IPC Server handler 6 on 54104] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33054 is added to blk_-6017619215766050746_1016{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:52910|RBW], ReplicaUnderConstruction[127.0.0.1:33054|RBW]]} size 0 2013-05-24 22:00:40,111 INFO [IPC Server handler 4 on 54104] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52910 is added to blk_-6017619215766050746_1016 size 42049 2013-05-24 22:00:40,119 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,42690,1369432806911] wal.HLogSplitter(602): Processed 472 edits across 3 regions; log file=hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997-splitting/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432829856 is corrupted = false progress failed = false 2013-05-24 22:00:40,152 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:51523-0x13ed88e2cdf0010 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432831597 2013-05-24 22:00:40,152 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:53632-0x13ed88e2cdf000f Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432831597 2013-05-24 22:00:40,153 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] regionserver.SplitLogWorker(451): successfully transitioned task /2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432831597 to final state DONE kiyo.gq1.ygridcore.net,51523,1369432814392 2013-05-24 22:00:40,153 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] regionserver.SplitLogWorker(385): worker kiyo.gq1.ygridcore.net,51523,1369432814392 done with task /2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432831597 in 305ms 2013-05-24 22:00:40,153 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392.logRoller] wal.FSHLog(522): Rolled WAL /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,51523,1369432814392/kiyo.gq1.ygridcore.net%2C51523%2C1369432814392.1369432839718, entries=1, filesize=41.1 K; new WAL=/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,51523,1369432814392/kiyo.gq1.ygridcore.net%2C51523%2C1369432814392.1369432840072 2013-05-24 22:00:40,160 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C49701%2C1369432806997-splitting%2Fkiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432829856 2013-05-24 22:00:40,160 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:42690-0x13ed88e2cdf0006 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C49701%2C1369432806997-splitting%2Fkiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432829856 2013-05-24 22:00:40,161 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(716): task /2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432831597 entered state: DONE kiyo.gq1.ygridcore.net,51523,1369432814392 2013-05-24 22:00:40,162 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,42690,1369432806911] regionserver.SplitLogWorker(451): successfully transitioned task /1/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C49701%2C1369432806997-splitting%2Fkiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432829856 to final state DONE kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:40,163 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,42690,1369432806911] regionserver.SplitLogWorker(385): worker kiyo.gq1.ygridcore.net,42690,1369432806911 done with task /1/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C49701%2C1369432806997-splitting%2Fkiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432829856 in 471ms 2013-05-24 22:00:40,163 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(716): task /1/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C49701%2C1369432806997-splitting%2Fkiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432829856 entered state: DONE kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:40,169 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] regionserver.SplitLogWorker(330): worker kiyo.gq1.ygridcore.net,51523,1369432814392 acquired task /2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432828321 2013-05-24 22:00:40,171 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] wal.HLogSplitter(506): Splitting hlog: hdfs://localhost:54104/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,39146,1369432814418-splitting/kiyo.gq1.ygridcore.net%2C39146%2C1369432814418.1369432828321, length=41030 2013-05-24 22:00:40,172 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] wal.HLogSplitter(507): DistributedLogReplay = true 2013-05-24 22:00:40,209 DEBUG [pool-1-thread-1-EventThread] wal.HLogSplitter(692): Archived processed log hdfs://localhost:54104/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,39146,1369432814418-splitting/kiyo.gq1.ygridcore.net%2C39146%2C1369432814418.1369432831597 to hdfs://localhost:54104/user/hortonzy/hbase/.oldlogs/kiyo.gq1.ygridcore.net%2C39146%2C1369432814418.1369432831597 2013-05-24 22:00:40,210 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(629): Done splitting /2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432831597 2013-05-24 22:00:40,211 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:53632-0x13ed88e2cdf000f Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432828321 2013-05-24 22:00:40,218 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,42690,1369432806911] regionserver.SplitLogWorker(330): worker kiyo.gq1.ygridcore.net,42690,1369432806911 acquired task /1/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C49701%2C1369432806997-splitting%2Fkiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432833549 2013-05-24 22:00:40,221 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,42690,1369432806911] wal.HLogSplitter(506): Splitting hlog: hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997-splitting/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432833549, length=23036 2013-05-24 22:00:40,221 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,42690,1369432806911] wal.HLogSplitter(507): DistributedLogReplay = true 2013-05-24 22:00:40,221 DEBUG [Thread-584] client.MetaScanner(160): Scanning .META. starting at row=test,sss,00000000000000 for max=10 with caching=10 2013-05-24 22:00:40,221 DEBUG [Thread-584] client.ClientScanner(96): Creating scanner over .META. starting at key 'test,sss,00000000000000' 2013-05-24 22:00:40,222 DEBUG [Thread-584] client.ClientScanner(209): Advancing internal scanner to startKey at 'test,sss,00000000000000' 2013-05-24 22:00:40,252 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:51523-0x13ed88e2cdf0010 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432828321 2013-05-24 22:00:40,253 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:53632-0x13ed88e2cdf000f Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432831597 2013-05-24 22:00:40,253 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] util.FSHDFSUtils(86): Recovering lease on dfs file hdfs://localhost:54104/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,39146,1369432814418-splitting/kiyo.gq1.ygridcore.net%2C39146%2C1369432814418.1369432828321 2013-05-24 22:00:40,253 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager$DeleteAsyncCallback(1529): deleted /2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432831597 2013-05-24 22:00:40,254 DEBUG [pool-1-thread-1-EventThread] wal.HLogSplitter(692): Archived processed log hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997-splitting/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432829856 to hdfs://localhost:54096/user/hortonzy/hbase/.oldlogs/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432829856 2013-05-24 22:00:40,254 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] util.FSHDFSUtils(156): recoverLease=true, attempt=0 on file=hdfs://localhost:54104/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,39146,1369432814418-splitting/kiyo.gq1.ygridcore.net%2C39146%2C1369432814418.1369432828321 after 1ms 2013-05-24 22:00:40,254 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(781): task /2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432828321 acquired by kiyo.gq1.ygridcore.net,51523,1369432814392 2013-05-24 22:00:40,254 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(629): Done splitting /1/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C49701%2C1369432806997-splitting%2Fkiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432829856 2013-05-24 22:00:40,255 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C49701%2C1369432806997-splitting%2Fkiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432833549 2013-05-24 22:00:40,258 DEBUG [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 41022, fileLength: 41030, trailerPresent: true 2013-05-24 22:00:40,266 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:42690-0x13ed88e2cdf0006 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C49701%2C1369432806997-splitting%2Fkiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432833549 2013-05-24 22:00:40,267 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,42690,1369432806911] util.FSHDFSUtils(86): Recovering lease on dfs file hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997-splitting/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432833549 2013-05-24 22:00:40,269 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,42690,1369432806911] util.FSHDFSUtils(156): recoverLease=true, attempt=0 on file=hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997-splitting/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432833549 after 2ms 2013-05-24 22:00:40,272 DEBUG [Thread-584] client.ClientScanner(196): Finished scanning region {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,} 2013-05-24 22:00:40,272 DEBUG [Thread-584] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,sss,1369432820320.872960a61a4ceeb9f64577d8b7cb0ddf. is kiyo.gq1.ygridcore.net:42690 2013-05-24 22:00:40,272 DEBUG [SplitLogWorker-kiyo.gq1.ygridcore.net,42690,1369432806911] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 23028, fileLength: 23036, trailerPresent: true 2013-05-24 22:00:40,283 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C49701%2C1369432806997-splitting%2Fkiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432829856 2013-05-24 22:00:40,283 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:42690-0x13ed88e2cdf0006 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/splitlog 2013-05-24 22:00:40,283 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] regionserver.SplitLogWorker(572): tasks arrived or departed 2013-05-24 22:00:40,283 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager$DeleteAsyncCallback(1529): deleted /1/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C49701%2C1369432806997-splitting%2Fkiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432829856 2013-05-24 22:00:40,285 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(781): task /1/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C49701%2C1369432806997-splitting%2Fkiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432833549 acquired by kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:40,287 DEBUG [WriterThread-0] wal.HLogSplitter$WriterThread(1102): Writer thread Thread[WriterThread-0,5,main]: starting 2013-05-24 22:00:40,287 DEBUG [WriterThread-2] wal.HLogSplitter$WriterThread(1102): Writer thread Thread[WriterThread-2,5,main]: starting 2013-05-24 22:00:40,287 DEBUG [WriterThread-1] wal.HLogSplitter$WriterThread(1102): Writer thread Thread[WriterThread-1,5,main]: starting 2013-05-24 22:00:40,287 DEBUG [WriterThread-2] wal.HLogSplitter$WriterThread(1102): Writer thread Thread[WriterThread-2,5,main]: starting 2013-05-24 22:00:40,287 DEBUG [WriterThread-0] wal.HLogSplitter$WriterThread(1102): Writer thread Thread[WriterThread-0,5,main]: starting 2013-05-24 22:00:40,287 DEBUG [WriterThread-1] wal.HLogSplitter$WriterThread(1102): Writer thread Thread[WriterThread-1,5,main]: starting 2013-05-24 22:00:40,293 INFO [WriterThread-1] zookeeper.RecoverableZooKeeper(119): Process identifier=hconnection-0x4e7493ae connecting to ZooKeeper ensemble=localhost:51411 2013-05-24 22:00:40,293 INFO [WriterThread-0] zookeeper.RecoverableZooKeeper(119): Process identifier=hconnection-0x478bacbf connecting to ZooKeeper ensemble=localhost:51411 2013-05-24 22:00:40,297 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] wal.HLogSplitter(595): Finishing writing output logs and closing down. 2013-05-24 22:00:40,297 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] wal.HLogSplitter$OutputSink(1258): Waiting for split writer threads to finish 2013-05-24 22:00:40,299 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,42690,1369432806911] wal.HLogSplitter(595): Finishing writing output logs and closing down. 2013-05-24 22:00:40,299 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,42690,1369432806911] wal.HLogSplitter$OutputSink(1258): Waiting for split writer threads to finish 2013-05-24 22:00:40,308 DEBUG [WriterThread-0-EventThread] zookeeper.ZooKeeperWatcher(307): hconnection-0x478bacbf Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2013-05-24 22:00:40,308 DEBUG [WriterThread-1-EventThread] zookeeper.ZooKeeperWatcher(307): hconnection-0x4e7493ae Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2013-05-24 22:00:40,309 DEBUG [WriterThread-0-EventThread] zookeeper.ZooKeeperWatcher(384): hconnection-0x478bacbf-0x13ed88e2cdf001f connected 2013-05-24 22:00:40,309 DEBUG [WriterThread-1-EventThread] zookeeper.ZooKeeperWatcher(384): hconnection-0x4e7493ae-0x13ed88e2cdf0020 connected 2013-05-24 22:00:40,315 DEBUG [WriterThread-1] client.MetaScanner(160): Scanning .META. starting at row=test,,00000000000000 for max=10 with caching=10 2013-05-24 22:00:40,315 DEBUG [WriterThread-0] client.MetaScanner(160): Scanning .META. starting at row=test,qqq,00000000000000 for max=10 with caching=10 2013-05-24 22:00:40,315 DEBUG [WriterThread-1] client.ClientScanner(96): Creating scanner over .META. starting at key 'test,,00000000000000' 2013-05-24 22:00:40,315 DEBUG [WriterThread-0] client.ClientScanner(96): Creating scanner over .META. starting at key 'test,qqq,00000000000000' 2013-05-24 22:00:40,315 DEBUG [WriterThread-1] client.ClientScanner(209): Advancing internal scanner to startKey at 'test,,00000000000000' 2013-05-24 22:00:40,315 DEBUG [WriterThread-0] client.ClientScanner(209): Advancing internal scanner to startKey at 'test,qqq,00000000000000' 2013-05-24 22:00:40,324 DEBUG [WriterThread-1] client.ClientScanner(196): Finished scanning region {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,} 2013-05-24 22:00:40,325 DEBUG [WriterThread-1] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,,1369432825000.45883ac87958f956040ff389c9f7eb86. is kiyo.gq1.ygridcore.net:51523 2013-05-24 22:00:40,326 DEBUG [WriterThread-0] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,qqq,1369432820320.ec0b9e20bb8d4dc8fee9b3e80c5cae32. is kiyo.gq1.ygridcore.net:42690 2013-05-24 22:00:40,326 DEBUG [WriterThread-0] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,rrr,1369432820320.85815417b0e1742517fac3960fac86c1. is kiyo.gq1.ygridcore.net:42690 2013-05-24 22:00:40,326 DEBUG [WriterThread-0] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,sss,1369432820320.872960a61a4ceeb9f64577d8b7cb0ddf. is kiyo.gq1.ygridcore.net:42690 2013-05-24 22:00:40,327 DEBUG [WriterThread-0] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,ttt,1369432820320.f53d942877419dd243e3fb7e46c2cf97. is kiyo.gq1.ygridcore.net:42690 2013-05-24 22:00:40,327 DEBUG [WriterThread-0] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,uuu,1369432820320.dba8b67ade91438cad8a5209736330e6. is kiyo.gq1.ygridcore.net:42690 2013-05-24 22:00:40,327 DEBUG [WriterThread-0] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,vvv,1369432820320.84d2bf1edb7e9df4777e80207fc739d2. is kiyo.gq1.ygridcore.net:42690 2013-05-24 22:00:40,327 DEBUG [WriterThread-0] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,www,1369432820320.643c76d94d70be44d35b5a12853bb852. is kiyo.gq1.ygridcore.net:42690 2013-05-24 22:00:40,328 DEBUG [WriterThread-0] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,xxx,1369432820320.b03c7b8b6013fa49f346edeb3a7cad87. is kiyo.gq1.ygridcore.net:42690 2013-05-24 22:00:40,328 DEBUG [WriterThread-0] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,yyy,1369432820321.acf78311e603cb0d02309ae08763d196. is kiyo.gq1.ygridcore.net:42690 2013-05-24 22:00:40,328 DEBUG [WriterThread-0] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,zzz,1369432820321.0b4d70d096cb6cbce3376acd563c77dd. is kiyo.gq1.ygridcore.net:42690 2013-05-24 22:00:40,349 DEBUG [WriterThread-0] wal.WALEditsReplaySink(128): number of rows:703 are sent by batch! spent 12(ms)! 2013-05-24 22:00:40,349 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,42690,1369432806911] wal.HLogSplitter$OutputSink(1276): Split writers finished 2013-05-24 22:00:40,351 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,42690,1369432806911] client.HConnectionManager$HConnectionImplementation(1640): Closing zookeeper sessionid=0x13ed88e2cdf001f 2013-05-24 22:00:40,390 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,42690,1369432806911] wal.HLogSplitter(602): Processed 235 edits across 1 regions; log file=hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997-splitting/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432833549 is corrupted = false progress failed = false 2013-05-24 22:00:40,428 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.logRoller] regionserver.LogRoller(90): HLog roll requested 2013-05-24 22:00:40,440 ERROR [IPC Server handler 4 on 54096] security.UserGroupInformation(1480): PriviledgedActionException as:hortonzy.hfs.0 (auth:SIMPLE) cause:java.io.FileNotFoundException: File does not exist: /user/hortonzy/hbase/.oldlogs/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432809419 2013-05-24 22:00:40,442 INFO [ReplicationExecutor-0.replicationSource,2-kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.ReplicationSource(501): NB dead servers : 1 2013-05-24 22:00:40,442 INFO [ReplicationExecutor-0.replicationSource,2-kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.ReplicationSource(511): Possible location hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432809419 2013-05-24 22:00:40,443 INFO [ReplicationExecutor-0.replicationSource,2-kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.ReplicationSource(511): Possible location hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997-splitting/kiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432809419 2013-05-24 22:00:40,444 INFO [ReplicationExecutor-0.replicationSource,2-kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.ReplicationSource(514): Log hdfs://localhost:54096/user/hortonzy/hbase/.oldlogs/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432809419 still exists at hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997-splitting/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432809419 2013-05-24 22:00:40,444 DEBUG [ReplicationExecutor-0.replicationSource,2-kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.ReplicationSource(583): Unable to open a reader, sleeping 100 times 5 2013-05-24 22:00:40,476 DEBUG [WriterThread-1] wal.WALEditsReplaySink(128): number of rows:1734 are sent by batch! spent 135(ms)! 2013-05-24 22:00:40,477 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] wal.HLogSplitter$OutputSink(1276): Split writers finished 2013-05-24 22:00:40,478 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] client.HConnectionManager$HConnectionImplementation(1640): Closing zookeeper sessionid=0x13ed88e2cdf0020 2013-05-24 22:00:40,493 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:42690-0x13ed88e2cdf0006 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C49701%2C1369432806997-splitting%2Fkiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432833549 2013-05-24 22:00:40,493 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C49701%2C1369432806997-splitting%2Fkiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432833549 2013-05-24 22:00:40,493 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,42690,1369432806911] regionserver.SplitLogWorker(451): successfully transitioned task /1/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C49701%2C1369432806997-splitting%2Fkiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432833549 to final state DONE kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:40,493 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,42690,1369432806911] regionserver.SplitLogWorker(385): worker kiyo.gq1.ygridcore.net,42690,1369432806911 done with task /1/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C49701%2C1369432806997-splitting%2Fkiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432833549 in 275ms 2013-05-24 22:00:40,546 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] wal.HLogSplitter(602): Processed 21 edits across 1 regions; log file=hdfs://localhost:54104/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,39146,1369432814418-splitting/kiyo.gq1.ygridcore.net%2C39146%2C1369432814418.1369432828321 is corrupted = false progress failed = false 2013-05-24 22:00:40,548 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(716): task /1/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C49701%2C1369432806997-splitting%2Fkiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432833549 entered state: DONE kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:40,576 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:51523-0x13ed88e2cdf0010 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432828321 2013-05-24 22:00:40,577 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:53632-0x13ed88e2cdf000f Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432828321 2013-05-24 22:00:40,577 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] regionserver.SplitLogWorker(451): successfully transitioned task /2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432828321 to final state DONE kiyo.gq1.ygridcore.net,51523,1369432814392 2013-05-24 22:00:40,577 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] regionserver.SplitLogWorker(385): worker kiyo.gq1.ygridcore.net,51523,1369432814392 done with task /2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432828321 in 407ms 2013-05-24 22:00:40,611 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,42690,1369432806911] regionserver.SplitLogWorker(330): worker kiyo.gq1.ygridcore.net,42690,1369432806911 acquired task /1/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C49701%2C1369432806997-splitting%2Fkiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432809419 2013-05-24 22:00:40,611 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(716): task /2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432828321 entered state: DONE kiyo.gq1.ygridcore.net,51523,1369432814392 2013-05-24 22:00:40,613 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,42690,1369432806911] wal.HLogSplitter(506): Splitting hlog: hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997-splitting/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432809419, length=68792 2013-05-24 22:00:40,613 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,42690,1369432806911] wal.HLogSplitter(507): DistributedLogReplay = true 2013-05-24 22:00:40,634 DEBUG [kiyo.gq1.ygridcore.net,53632,1369432814258.splitLogManagerTimeoutMonitor] master.SplitLogManager$TimeoutMonitor(1337): total tasks = 4 unassigned = 3 2013-05-24 22:00:41,257 DEBUG [kiyo.gq1.ygridcore.net,49101,1369432806564.splitLogManagerTimeoutMonitor] master.SplitLogManager$TimeoutMonitor(1337): total tasks = 3 unassigned = 2 2013-05-24 22:00:41,258 ERROR [IPC Server handler 6 on 54096] security.UserGroupInformation(1480): PriviledgedActionException as:hortonzy.hfs.0 (auth:SIMPLE) cause:java.io.FileNotFoundException: File does not exist: /user/hortonzy/hbase/.oldlogs/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432809419 2013-05-24 22:00:41,258 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.logRoller] wal.FSHLog(658): cleanupCurrentWriter waiting for transactions to get synced total 2465 synced till here 2464 2013-05-24 22:00:41,260 INFO [ReplicationExecutor-0.replicationSource,2-kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.ReplicationSource(501): NB dead servers : 1 2013-05-24 22:00:41,260 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] regionserver.SplitLogWorker(330): worker kiyo.gq1.ygridcore.net,51523,1369432814392 acquired task /2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432832288 2013-05-24 22:00:41,261 INFO [ReplicationExecutor-0.replicationSource,2-kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.ReplicationSource(511): Possible location hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432809419 2013-05-24 22:00:41,265 INFO [IPC Server handler 5 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53333 is added to blk_-4934693909141596598_1043{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:59993|RBW], ReplicaUnderConstruction[127.0.0.1:53333|RBW]]} size 0 2013-05-24 22:00:41,265 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] wal.HLogSplitter(506): Splitting hlog: hdfs://localhost:54104/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,39146,1369432814418-splitting/kiyo.gq1.ygridcore.net%2C39146%2C1369432814418.1369432832288, length=49811 2013-05-24 22:00:41,266 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432834215 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:41,266 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] wal.HLogSplitter(507): DistributedLogReplay = true 2013-05-24 22:00:41,266 INFO [IPC Server handler 1 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59993 is added to blk_-4934693909141596598_1043{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:59993|RBW], ReplicaUnderConstruction[127.0.0.1:53333|RBW]]} size 0 2013-05-24 22:00:41,267 INFO [ReplicationExecutor-0.replicationSource,2-kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.ReplicationSource(511): Possible location hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997-splitting/kiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432809419 2013-05-24 22:00:41,268 INFO [ReplicationExecutor-0.replicationSource,2-kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.ReplicationSource(514): Log hdfs://localhost:54096/user/hortonzy/hbase/.oldlogs/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432809419 still exists at hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997-splitting/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432809419 2013-05-24 22:00:41,268 DEBUG [ReplicationExecutor-0.replicationSource,2-kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.ReplicationSource(583): Unable to open a reader, sleeping 100 times 6 2013-05-24 22:00:41,268 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 44122, fileLength: 44130, trailerPresent: true 2013-05-24 22:00:41,309 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:42690-0x13ed88e2cdf0006 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C49701%2C1369432806997-splitting%2Fkiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432809419 2013-05-24 22:00:41,309 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,42690,1369432806911] util.FSHDFSUtils(86): Recovering lease on dfs file hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997-splitting/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432809419 2013-05-24 22:00:41,310 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,42690,1369432806911] util.FSHDFSUtils(156): recoverLease=true, attempt=0 on file=hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997-splitting/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432809419 after 0ms 2013-05-24 22:00:41,314 DEBUG [SplitLogWorker-kiyo.gq1.ygridcore.net,42690,1369432806911] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 68784, fileLength: 68792, trailerPresent: true 2013-05-24 22:00:41,341 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:51523-0x13ed88e2cdf0010 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432832288 2013-05-24 22:00:41,342 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] util.FSHDFSUtils(86): Recovering lease on dfs file hdfs://localhost:54104/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,39146,1369432814418-splitting/kiyo.gq1.ygridcore.net%2C39146%2C1369432814418.1369432832288 2013-05-24 22:00:41,343 DEBUG [pool-1-thread-1-EventThread] wal.HLogSplitter(692): Archived processed log hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997-splitting/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432833549 to hdfs://localhost:54096/user/hortonzy/hbase/.oldlogs/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432833549 2013-05-24 22:00:41,343 DEBUG [pool-1-thread-1-EventThread] wal.HLogSplitter(692): Archived processed log hdfs://localhost:54104/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,39146,1369432814418-splitting/kiyo.gq1.ygridcore.net%2C39146%2C1369432814418.1369432828321 to hdfs://localhost:54104/user/hortonzy/hbase/.oldlogs/kiyo.gq1.ygridcore.net%2C39146%2C1369432814418.1369432828321 2013-05-24 22:00:41,343 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] util.FSHDFSUtils(156): recoverLease=true, attempt=0 on file=hdfs://localhost:54104/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,39146,1369432814418-splitting/kiyo.gq1.ygridcore.net%2C39146%2C1369432814418.1369432832288 after 1ms 2013-05-24 22:00:41,344 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(629): Done splitting /1/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C49701%2C1369432806997-splitting%2Fkiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432833549 2013-05-24 22:00:41,344 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C49701%2C1369432806997-splitting%2Fkiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432809419 2013-05-24 22:00:41,344 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(629): Done splitting /2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432828321 2013-05-24 22:00:41,344 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:53632-0x13ed88e2cdf000f Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432832288 2013-05-24 22:00:41,347 DEBUG [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 49803, fileLength: 49811, trailerPresent: true 2013-05-24 22:00:41,403 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.logRoller] wal.FSHLog(522): Rolled WAL /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432839980, entries=345, filesize=60.0 K; new WAL=/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432840428 2013-05-24 22:00:41,518 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C49701%2C1369432806997-splitting%2Fkiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432833549 2013-05-24 22:00:41,518 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:53632-0x13ed88e2cdf000f Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432828321 2013-05-24 22:00:41,518 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager$DeleteAsyncCallback(1529): deleted /1/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C49701%2C1369432806997-splitting%2Fkiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432833549 2013-05-24 22:00:41,519 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager$DeleteAsyncCallback(1529): deleted /2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432828321 2013-05-24 22:00:41,519 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(781): task /2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432832288 acquired by kiyo.gq1.ygridcore.net,51523,1369432814392 2013-05-24 22:00:41,519 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(781): task /1/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C49701%2C1369432806997-splitting%2Fkiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432809419 acquired by kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:41,536 DEBUG [WriterThread-0] wal.HLogSplitter$WriterThread(1102): Writer thread Thread[WriterThread-0,5,main]: starting 2013-05-24 22:00:41,536 DEBUG [WriterThread-2] wal.HLogSplitter$WriterThread(1102): Writer thread Thread[WriterThread-2,5,main]: starting 2013-05-24 22:00:41,536 DEBUG [WriterThread-2] wal.HLogSplitter$WriterThread(1102): Writer thread Thread[WriterThread-2,5,main]: starting 2013-05-24 22:00:41,536 DEBUG [WriterThread-0] wal.HLogSplitter$WriterThread(1102): Writer thread Thread[WriterThread-0,5,main]: starting 2013-05-24 22:00:41,536 DEBUG [WriterThread-1] wal.HLogSplitter$WriterThread(1102): Writer thread Thread[WriterThread-1,5,main]: starting 2013-05-24 22:00:41,536 DEBUG [WriterThread-1] wal.HLogSplitter$WriterThread(1102): Writer thread Thread[WriterThread-1,5,main]: starting 2013-05-24 22:00:41,606 INFO [WriterThread-1] zookeeper.RecoverableZooKeeper(119): Process identifier=hconnection-0x115e01c2 connecting to ZooKeeper ensemble=localhost:51411 2013-05-24 22:00:41,608 INFO [WriterThread-1] zookeeper.RecoverableZooKeeper(119): Process identifier=hconnection-0x405a6d04 connecting to ZooKeeper ensemble=localhost:51411 2013-05-24 22:00:41,611 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] wal.HLogSplitter(595): Finishing writing output logs and closing down. 2013-05-24 22:00:41,611 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] wal.HLogSplitter$OutputSink(1258): Waiting for split writer threads to finish 2013-05-24 22:00:41,634 DEBUG [kiyo.gq1.ygridcore.net,53632,1369432814258.splitLogManagerTimeoutMonitor] master.SplitLogManager$TimeoutMonitor(1337): total tasks = 3 unassigned = 2 2013-05-24 22:00:41,718 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(583): Nothing to replicate, sleeping 100 times 10 2013-05-24 22:00:41,725 DEBUG [WriterThread-1-EventThread] zookeeper.ZooKeeperWatcher(307): hconnection-0x115e01c2 Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2013-05-24 22:00:41,726 DEBUG [WriterThread-1-EventThread] zookeeper.ZooKeeperWatcher(307): hconnection-0x405a6d04 Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2013-05-24 22:00:41,727 DEBUG [WriterThread-1-EventThread] zookeeper.ZooKeeperWatcher(384): hconnection-0x115e01c2-0x13ed88e2cdf0021 connected 2013-05-24 22:00:41,727 DEBUG [WriterThread-1-EventThread] zookeeper.ZooKeeperWatcher(384): hconnection-0x405a6d04-0x13ed88e2cdf0022 connected 2013-05-24 22:00:41,736 DEBUG [WriterThread-1] client.MetaScanner(160): Scanning .META. starting at row=test,,00000000000000 for max=10 with caching=10 2013-05-24 22:00:41,737 DEBUG [WriterThread-1] client.ClientScanner(96): Creating scanner over .META. starting at key 'test,,00000000000000' 2013-05-24 22:00:41,737 DEBUG [WriterThread-1] client.ClientScanner(209): Advancing internal scanner to startKey at 'test,,00000000000000' 2013-05-24 22:00:41,737 DEBUG [WriterThread-1] client.MetaScanner(160): Scanning .META. starting at row=test,,00000000000000 for max=10 with caching=10 2013-05-24 22:00:41,737 DEBUG [WriterThread-1] client.ClientScanner(96): Creating scanner over .META. starting at key 'test,,00000000000000' 2013-05-24 22:00:41,737 DEBUG [WriterThread-1] client.ClientScanner(209): Advancing internal scanner to startKey at 'test,,00000000000000' 2013-05-24 22:00:41,742 DEBUG [WriterThread-1] client.ClientScanner(196): Finished scanning region {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,} 2013-05-24 22:00:41,742 DEBUG [WriterThread-1] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,,1369432825000.45883ac87958f956040ff389c9f7eb86. is kiyo.gq1.ygridcore.net:51523 2013-05-24 22:00:41,746 DEBUG [WriterThread-1] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,,1369432820315.a88e8c9e73f832adb3635d6996d9e72d. is kiyo.gq1.ygridcore.net:42690 2013-05-24 22:00:41,746 DEBUG [WriterThread-1] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,bbb,1369432820319.6a4e618fc444a41a507e36bb35c860a6. is kiyo.gq1.ygridcore.net:42690 2013-05-24 22:00:41,746 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,42690,1369432806911] wal.HLogSplitter(595): Finishing writing output logs and closing down. 2013-05-24 22:00:41,746 DEBUG [WriterThread-1] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,ccc,1369432820319.117d9d6b052965f7196396ea21f07a61. is kiyo.gq1.ygridcore.net:42690 2013-05-24 22:00:41,746 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,42690,1369432806911] wal.HLogSplitter$OutputSink(1258): Waiting for split writer threads to finish 2013-05-24 22:00:41,747 DEBUG [WriterThread-1] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,ddd,1369432820319.925080806d078451834e3417849dd4a9. is kiyo.gq1.ygridcore.net:42690 2013-05-24 22:00:41,747 DEBUG [WriterThread-1] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,eee,1369432820319.6f0a98b71d64131085456015699d44f0. is kiyo.gq1.ygridcore.net:42690 2013-05-24 22:00:41,747 DEBUG [WriterThread-1] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,fff,1369432820319.f852f52d87b0b7acd213f5b49074dde6. is kiyo.gq1.ygridcore.net:42690 2013-05-24 22:00:41,748 DEBUG [WriterThread-1] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,ggg,1369432820319.4b3ff3efde12c5afe1220e81d155ac73. is kiyo.gq1.ygridcore.net:42690 2013-05-24 22:00:41,748 DEBUG [WriterThread-1] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,hhh,1369432820320.a9a8fa0d1fb6a035656bc264ec313d2b. is kiyo.gq1.ygridcore.net:42690 2013-05-24 22:00:41,748 DEBUG [WriterThread-1] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,iii,1369432820320.992fe9e8d6a399ebc508d777668f74ad. is kiyo.gq1.ygridcore.net:42690 2013-05-24 22:00:41,749 DEBUG [WriterThread-1] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,jjj,1369432820320.eb38c552d2a930aa1ce862202da80908. is kiyo.gq1.ygridcore.net:42690 2013-05-24 22:00:41,769 DEBUG [WriterThread-1] wal.WALEditsReplaySink(128): number of rows:23 are sent by batch! spent 4(ms)! 2013-05-24 22:00:41,775 DEBUG [WriterThread-2] wal.WALEditsReplaySink(128): number of rows:82 are sent by batch! spent 6(ms)! 2013-05-24 22:00:41,809 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392.logRoller] regionserver.LogRoller(90): HLog roll requested 2013-05-24 22:00:41,810 DEBUG [WriterThread-1] wal.WALEditsReplaySink(128): number of rows:2127 are sent by batch! spent 37(ms)! 2013-05-24 22:00:41,810 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] wal.HLogSplitter$OutputSink(1276): Split writers finished 2013-05-24 22:00:41,811 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] client.HConnectionManager$HConnectionImplementation(1640): Closing zookeeper sessionid=0x13ed88e2cdf0021 2013-05-24 22:00:41,823 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.logRoller] regionserver.LogRoller(90): HLog roll requested 2013-05-24 22:00:41,830 DEBUG [WriterThread-0] wal.WALEditsReplaySink(128): number of rows:2000 are sent by batch! spent 64(ms)! 2013-05-24 22:00:41,831 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,42690,1369432806911] wal.HLogSplitter$OutputSink(1276): Split writers finished 2013-05-24 22:00:41,831 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,42690,1369432806911] client.HConnectionManager$HConnectionImplementation(1640): Closing zookeeper sessionid=0x13ed88e2cdf0022 2013-05-24 22:00:41,869 ERROR [IPC Server handler 9 on 54096] security.UserGroupInformation(1480): PriviledgedActionException as:hortonzy.hfs.0 (auth:SIMPLE) cause:java.io.FileNotFoundException: File does not exist: /user/hortonzy/hbase/.oldlogs/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432809419 2013-05-24 22:00:41,870 INFO [ReplicationExecutor-0.replicationSource,2-kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.ReplicationSource(501): NB dead servers : 1 2013-05-24 22:00:41,871 INFO [ReplicationExecutor-0.replicationSource,2-kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.ReplicationSource(511): Possible location hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432809419 2013-05-24 22:00:41,872 INFO [ReplicationExecutor-0.replicationSource,2-kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.ReplicationSource(511): Possible location hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997-splitting/kiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432809419 2013-05-24 22:00:41,872 INFO [ReplicationExecutor-0.replicationSource,2-kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.ReplicationSource(514): Log hdfs://localhost:54096/user/hortonzy/hbase/.oldlogs/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432809419 still exists at hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997-splitting/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432809419 2013-05-24 22:00:41,873 DEBUG [ReplicationExecutor-0.replicationSource,2-kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.ReplicationSource(583): Unable to open a reader, sleeping 100 times 7 2013-05-24 22:00:42,257 DEBUG [kiyo.gq1.ygridcore.net,49101,1369432806564.splitLogManagerTimeoutMonitor] master.SplitLogManager$TimeoutMonitor(1337): total tasks = 2 unassigned = 1 2013-05-24 22:00:42,316 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] wal.HLogSplitter(602): Processed 16 edits across 1 regions; log file=hdfs://localhost:54104/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,39146,1369432814418-splitting/kiyo.gq1.ygridcore.net%2C39146%2C1369432814418.1369432832288 is corrupted = false progress failed = false 2013-05-24 22:00:42,343 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,42690,1369432806911] wal.HLogSplitter(602): Processed 703 edits across 3 regions; log file=hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997-splitting/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432809419 is corrupted = false progress failed = false 2013-05-24 22:00:42,376 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:51523-0x13ed88e2cdf0010 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432832288 2013-05-24 22:00:42,376 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:53632-0x13ed88e2cdf000f Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432832288 2013-05-24 22:00:42,376 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] regionserver.SplitLogWorker(451): successfully transitioned task /2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432832288 to final state DONE kiyo.gq1.ygridcore.net,51523,1369432814392 2013-05-24 22:00:42,376 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] regionserver.SplitLogWorker(385): worker kiyo.gq1.ygridcore.net,51523,1369432814392 done with task /2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432832288 in 1116ms 2013-05-24 22:00:42,471 INFO [IPC Server handler 0 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53333 is added to blk_5040236901299817573_1044{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:59993|RBW], ReplicaUnderConstruction[127.0.0.1:53333|RBW]]} size 0 2013-05-24 22:00:42,472 INFO [IPC Server handler 6 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59993 is added to blk_5040236901299817573_1044 size 132243 2013-05-24 22:00:42,488 INFO [IPC Server handler 9 on 54104] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33054 is added to blk_3860213949743194887_1017{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:52910|RBW], ReplicaUnderConstruction[127.0.0.1:33054|RBW]]} size 0 2013-05-24 22:00:42,490 INFO [IPC Server handler 3 on 54104] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52910 is added to blk_3860213949743194887_1017 size 88945 2013-05-24 22:00:42,524 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C49701%2C1369432806997-splitting%2Fkiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432809419 2013-05-24 22:00:42,524 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(716): task /2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432832288 entered state: DONE kiyo.gq1.ygridcore.net,51523,1369432814392 2013-05-24 22:00:42,524 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:42690-0x13ed88e2cdf0006 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C49701%2C1369432806997-splitting%2Fkiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432809419 2013-05-24 22:00:42,525 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,42690,1369432806911] regionserver.SplitLogWorker(451): successfully transitioned task /1/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C49701%2C1369432806997-splitting%2Fkiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432809419 to final state DONE kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:42,526 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,42690,1369432806911] regionserver.SplitLogWorker(385): worker kiyo.gq1.ygridcore.net,42690,1369432806911 done with task /1/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C49701%2C1369432806997-splitting%2Fkiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432809419 in 1915ms 2013-05-24 22:00:42,554 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.logRoller] wal.FSHLog(522): Rolled WAL /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432840428, entries=859, filesize=129.1 K; new WAL=/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432841823 2013-05-24 22:00:42,564 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392.logRoller] wal.FSHLog(522): Rolled WAL /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,51523,1369432814392/kiyo.gq1.ygridcore.net%2C51523%2C1369432814392.1369432840072, entries=2, filesize=86.9 K; new WAL=/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,51523,1369432814392/kiyo.gq1.ygridcore.net%2C51523%2C1369432814392.1369432841809 2013-05-24 22:00:42,571 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(716): task /1/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C49701%2C1369432806997-splitting%2Fkiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432809419 entered state: DONE kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:42,571 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] regionserver.SplitLogWorker(330): worker kiyo.gq1.ygridcore.net,51523,1369432814392 acquired task /2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432818197 2013-05-24 22:00:42,572 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] wal.HLogSplitter(506): Splitting hlog: hdfs://localhost:54104/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,39146,1369432814418-splitting/kiyo.gq1.ygridcore.net%2C39146%2C1369432814418.1369432818197, length=46425 2013-05-24 22:00:42,572 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] wal.HLogSplitter(507): DistributedLogReplay = true 2013-05-24 22:00:42,573 ERROR [IPC Server handler 5 on 54096] security.UserGroupInformation(1480): PriviledgedActionException as:hortonzy.hfs.0 (auth:SIMPLE) cause:java.io.FileNotFoundException: File does not exist: /user/hortonzy/hbase/.oldlogs/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432809419 2013-05-24 22:00:42,577 INFO [ReplicationExecutor-0.replicationSource,2-kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.ReplicationSource(501): NB dead servers : 1 2013-05-24 22:00:42,578 INFO [ReplicationExecutor-0.replicationSource,2-kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.ReplicationSource(511): Possible location hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432809419 2013-05-24 22:00:42,578 INFO [ReplicationExecutor-0.replicationSource,2-kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.ReplicationSource(511): Possible location hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997-splitting/kiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432809419 2013-05-24 22:00:42,579 WARN [ReplicationExecutor-0.replicationSource,2-kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.ReplicationSource(547): 2-kiyo.gq1.ygridcore.net,49701,1369432806997 Got: java.io.IOException: File from recovered queue is nowhere to be found at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:528) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:301) Caused by: java.io.FileNotFoundException: File does not exist: /user/hortonzy/hbase/.oldlogs/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432809419 at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:39) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1370) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1319) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1299) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1273) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:425) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:241) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:41888) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:507) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1029) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1839) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1835) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1477) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1833) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73) at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:996) at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:984) at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:239) at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:206) at org.apache.hadoop.hdfs.DFSInputStream.(DFSInputStream.java:199) at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1141) at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:247) at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:80) at org.apache.hadoop.fs.FilterFileSystem.open(FilterFileSystem.java:155) at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:748) at org.apache.hadoop.hbase.regionserver.wal.HLogFactory.createReader(HLogFactory.java:116) at org.apache.hadoop.hbase.regionserver.wal.HLogFactory.createReader(HLogFactory.java:89) at org.apache.hadoop.hbase.regionserver.wal.HLogFactory.createReader(HLogFactory.java:77) at org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.openReader(ReplicationHLogReaderManager.java:68) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:494) ... 1 more Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /user/hortonzy/hbase/.oldlogs/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432809419 at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:39) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1370) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1319) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1299) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1273) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:425) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:241) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:41888) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:507) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1029) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1839) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1835) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1477) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1833) at org.apache.hadoop.ipc.Client.call(Client.java:1301) at org.apache.hadoop.ipc.Client.call(Client.java:1253) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:204) at $Proxy15.getBlockLocations(Unknown Source) at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83) at $Proxy15.getBlockLocations(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:178) at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:266) at $Proxy19.getBlockLocations(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:994) ... 15 more 2013-05-24 22:00:42,580 DEBUG [ReplicationExecutor-0.replicationSource,2-kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.ReplicationSource(583): Unable to open a reader, sleeping 100 times 8 2013-05-24 22:00:42,613 DEBUG [pool-1-thread-1-EventThread] wal.HLogSplitter(692): Archived processed log hdfs://localhost:54104/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,39146,1369432814418-splitting/kiyo.gq1.ygridcore.net%2C39146%2C1369432814418.1369432832288 to hdfs://localhost:54104/user/hortonzy/hbase/.oldlogs/kiyo.gq1.ygridcore.net%2C39146%2C1369432814418.1369432832288 2013-05-24 22:00:42,614 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(629): Done splitting /2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432832288 2013-05-24 22:00:42,614 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:53632-0x13ed88e2cdf000f Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432818197 2013-05-24 22:00:42,632 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,42690,1369432806911] regionserver.SplitLogWorker(330): worker kiyo.gq1.ygridcore.net,42690,1369432806911 acquired task /1/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C49701%2C1369432806997-splitting%2Fkiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432832342 2013-05-24 22:00:42,632 DEBUG [pool-1-thread-1-EventThread] wal.HLogSplitter(692): Archived processed log hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997-splitting/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432809419 to hdfs://localhost:54096/user/hortonzy/hbase/.oldlogs/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432809419 2013-05-24 22:00:42,633 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(629): Done splitting /1/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C49701%2C1369432806997-splitting%2Fkiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432809419 2013-05-24 22:00:42,633 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,42690,1369432806911] wal.HLogSplitter(506): Splitting hlog: hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997-splitting/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432832342, length=69068 2013-05-24 22:00:42,633 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,42690,1369432806911] wal.HLogSplitter(507): DistributedLogReplay = true 2013-05-24 22:00:42,634 DEBUG [kiyo.gq1.ygridcore.net,53632,1369432814258.splitLogManagerTimeoutMonitor] master.SplitLogManager$TimeoutMonitor(1337): total tasks = 3 unassigned = 2 2013-05-24 22:00:42,708 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:51523-0x13ed88e2cdf0010 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432818197 2013-05-24 22:00:42,708 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:53632-0x13ed88e2cdf000f Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432832288 2013-05-24 22:00:42,710 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C49701%2C1369432806997-splitting%2Fkiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432832342 2013-05-24 22:00:42,710 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] util.FSHDFSUtils(86): Recovering lease on dfs file hdfs://localhost:54104/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,39146,1369432814418-splitting/kiyo.gq1.ygridcore.net%2C39146%2C1369432814418.1369432818197 2013-05-24 22:00:42,710 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager$DeleteAsyncCallback(1529): deleted /2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432832288 2013-05-24 22:00:42,711 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] util.FSHDFSUtils(156): recoverLease=true, attempt=0 on file=hdfs://localhost:54104/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,39146,1369432814418-splitting/kiyo.gq1.ygridcore.net%2C39146%2C1369432814418.1369432818197 after 1ms 2013-05-24 22:00:42,712 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(781): task /2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432818197 acquired by kiyo.gq1.ygridcore.net,51523,1369432814392 2013-05-24 22:00:42,715 DEBUG [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 46417, fileLength: 46425, trailerPresent: true 2013-05-24 22:00:42,720 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432834215 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:42,723 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 44122, fileLength: 44130, trailerPresent: true 2013-05-24 22:00:42,731 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432839980 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:42,734 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 61479, fileLength: 61487, trailerPresent: true 2013-05-24 22:00:42,739 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:42690-0x13ed88e2cdf0006 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C49701%2C1369432806997-splitting%2Fkiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432832342 2013-05-24 22:00:42,740 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,42690,1369432806911] util.FSHDFSUtils(86): Recovering lease on dfs file hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997-splitting/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432832342 2013-05-24 22:00:42,741 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,42690,1369432806911] util.FSHDFSUtils(156): recoverLease=true, attempt=0 on file=hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997-splitting/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432832342 after 1ms 2013-05-24 22:00:42,744 DEBUG [SplitLogWorker-kiyo.gq1.ygridcore.net,42690,1369432806911] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 69060, fileLength: 69068, trailerPresent: true 2013-05-24 22:00:42,752 WARN [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(685): Can't replicate because of a local or network error: java.io.IOException: Call to kiyo.gq1.ygridcore.net/68.142.244.61:39146 failed on local exception: java.io.EOFException at org.apache.hadoop.hbase.ipc.RpcClient.wrapException(RpcClient.java:1367) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1339) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1539) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1596) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.replicateWALEntry(AdminProtos.java:15111) at org.apache.hadoop.hbase.protobuf.ReplicationProtbufUtil.replicateWALEntry(ReplicationProtbufUtil.java:87) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.shipEdits(ReplicationSource.java:646) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:378) Caused by: java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:375) at org.apache.hadoop.hbase.ipc.RpcClient$Connection.readResponse(RpcClient.java:1003) at org.apache.hadoop.hbase.ipc.RpcClient$Connection.run(RpcClient.java:673) 2013-05-24 22:00:42,756 INFO [Thread-839] regionserver.ReplicationSource$2(804): Slave cluster looks down: Connection refused 2013-05-24 22:00:42,772 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C49701%2C1369432806997-splitting%2Fkiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432809419 2013-05-24 22:00:42,773 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager$DeleteAsyncCallback(1529): deleted /1/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C49701%2C1369432806997-splitting%2Fkiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432809419 2013-05-24 22:00:42,776 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(781): task /1/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C49701%2C1369432806997-splitting%2Fkiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432832342 acquired by kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:42,777 DEBUG [WriterThread-0] wal.HLogSplitter$WriterThread(1102): Writer thread Thread[WriterThread-0,5,main]: starting 2013-05-24 22:00:42,777 DEBUG [WriterThread-2] wal.HLogSplitter$WriterThread(1102): Writer thread Thread[WriterThread-2,5,main]: starting 2013-05-24 22:00:42,777 DEBUG [WriterThread-1] wal.HLogSplitter$WriterThread(1102): Writer thread Thread[WriterThread-1,5,main]: starting 2013-05-24 22:00:42,779 DEBUG [WriterThread-0] wal.HLogSplitter$WriterThread(1102): Writer thread Thread[WriterThread-0,5,main]: starting 2013-05-24 22:00:42,779 DEBUG [WriterThread-1] wal.HLogSplitter$WriterThread(1102): Writer thread Thread[WriterThread-1,5,main]: starting 2013-05-24 22:00:42,779 DEBUG [WriterThread-2] wal.HLogSplitter$WriterThread(1102): Writer thread Thread[WriterThread-2,5,main]: starting 2013-05-24 22:00:42,782 INFO [WriterThread-1] zookeeper.RecoverableZooKeeper(119): Process identifier=hconnection-0x3357f40a connecting to ZooKeeper ensemble=localhost:51411 2013-05-24 22:00:42,784 INFO [WriterThread-2] zookeeper.RecoverableZooKeeper(119): Process identifier=hconnection-0x398e54e8 connecting to ZooKeeper ensemble=localhost:51411 2013-05-24 22:00:42,785 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] wal.HLogSplitter(595): Finishing writing output logs and closing down. 2013-05-24 22:00:42,785 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] wal.HLogSplitter$OutputSink(1258): Waiting for split writer threads to finish 2013-05-24 22:00:42,791 DEBUG [WriterThread-1-EventThread] zookeeper.ZooKeeperWatcher(307): hconnection-0x3357f40a Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2013-05-24 22:00:42,792 DEBUG [WriterThread-1-EventThread] zookeeper.ZooKeeperWatcher(384): hconnection-0x3357f40a-0x13ed88e2cdf0023 connected 2013-05-24 22:00:42,805 DEBUG [WriterThread-2-EventThread] zookeeper.ZooKeeperWatcher(307): hconnection-0x398e54e8 Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2013-05-24 22:00:42,807 DEBUG [WriterThread-2-EventThread] zookeeper.ZooKeeperWatcher(384): hconnection-0x398e54e8-0x13ed88e2cdf0024 connected 2013-05-24 22:00:42,816 DEBUG [WriterThread-2] client.MetaScanner(160): Scanning .META. starting at row=test,,00000000000000 for max=10 with caching=10 2013-05-24 22:00:42,816 DEBUG [WriterThread-0] client.MetaScanner(160): Scanning .META. starting at row=test,nnn,00000000000000 for max=10 with caching=10 2013-05-24 22:00:42,816 DEBUG [WriterThread-2] client.ClientScanner(96): Creating scanner over .META. starting at key 'test,,00000000000000' 2013-05-24 22:00:42,817 DEBUG [WriterThread-0] client.ClientScanner(96): Creating scanner over .META. starting at key 'test,nnn,00000000000000' 2013-05-24 22:00:42,817 DEBUG [WriterThread-2] client.ClientScanner(209): Advancing internal scanner to startKey at 'test,,00000000000000' 2013-05-24 22:00:42,817 DEBUG [WriterThread-0] client.ClientScanner(209): Advancing internal scanner to startKey at 'test,nnn,00000000000000' 2013-05-24 22:00:42,822 DEBUG [WriterThread-2] client.ClientScanner(196): Finished scanning region {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,} 2013-05-24 22:00:42,823 DEBUG [WriterThread-2] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,,1369432825000.45883ac87958f956040ff389c9f7eb86. is kiyo.gq1.ygridcore.net:51523 2013-05-24 22:00:42,823 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,42690,1369432806911] wal.HLogSplitter(595): Finishing writing output logs and closing down. 2013-05-24 22:00:42,823 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,42690,1369432806911] wal.HLogSplitter$OutputSink(1258): Waiting for split writer threads to finish 2013-05-24 22:00:42,823 DEBUG [WriterThread-0] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,nnn,1369432820320.53e3aef4cc8815dd31923659bff5ca21. is kiyo.gq1.ygridcore.net:42690 2013-05-24 22:00:42,824 DEBUG [WriterThread-0] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,ooo,1369432820320.0d6df81a73ad3b2a79add80716da62ac. is kiyo.gq1.ygridcore.net:42690 2013-05-24 22:00:42,824 DEBUG [WriterThread-0] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,ppp,1369432820320.da89687648e34409286fd02380093aed. is kiyo.gq1.ygridcore.net:42690 2013-05-24 22:00:42,824 DEBUG [WriterThread-0] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,qqq,1369432820320.ec0b9e20bb8d4dc8fee9b3e80c5cae32. is kiyo.gq1.ygridcore.net:42690 2013-05-24 22:00:42,824 DEBUG [WriterThread-0] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,rrr,1369432820320.85815417b0e1742517fac3960fac86c1. is kiyo.gq1.ygridcore.net:42690 2013-05-24 22:00:42,825 DEBUG [WriterThread-0] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,sss,1369432820320.872960a61a4ceeb9f64577d8b7cb0ddf. is kiyo.gq1.ygridcore.net:42690 2013-05-24 22:00:42,825 DEBUG [WriterThread-0] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,ttt,1369432820320.f53d942877419dd243e3fb7e46c2cf97. is kiyo.gq1.ygridcore.net:42690 2013-05-24 22:00:42,825 DEBUG [WriterThread-0] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,uuu,1369432820320.dba8b67ade91438cad8a5209736330e6. is kiyo.gq1.ygridcore.net:42690 2013-05-24 22:00:42,825 DEBUG [WriterThread-0] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,vvv,1369432820320.84d2bf1edb7e9df4777e80207fc739d2. is kiyo.gq1.ygridcore.net:42690 2013-05-24 22:00:42,826 DEBUG [WriterThread-0] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,www,1369432820320.643c76d94d70be44d35b5a12853bb852. is kiyo.gq1.ygridcore.net:42690 2013-05-24 22:00:42,838 DEBUG [WriterThread-1] client.MetaScanner(160): Scanning .META. starting at row=test,mmm,00000000000000 for max=10 with caching=10 2013-05-24 22:00:42,838 DEBUG [WriterThread-1] client.ClientScanner(96): Creating scanner over .META. starting at key 'test,mmm,00000000000000' 2013-05-24 22:00:42,838 DEBUG [WriterThread-1] client.ClientScanner(209): Advancing internal scanner to startKey at 'test,mmm,00000000000000' 2013-05-24 22:00:42,844 DEBUG [WriterThread-1] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,mmm,1369432820320.ab7b9b9773243cff0ac09f3c379e33dc. is kiyo.gq1.ygridcore.net:42690 2013-05-24 22:00:42,850 DEBUG [WriterThread-0] wal.WALEditsReplaySink(128): number of rows:121 are sent by batch! spent 4(ms)! 2013-05-24 22:00:42,854 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(583): Since we are unable to replicate, sleeping 100 times 1 2013-05-24 22:00:42,894 DEBUG [WriterThread-0] wal.WALEditsReplaySink(128): number of rows:3 are sent by batch! spent 3(ms)! 2013-05-24 22:00:42,898 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.logRoller] regionserver.LogRoller(90): HLog roll requested 2013-05-24 22:00:42,900 DEBUG [WriterThread-2] wal.WALEditsReplaySink(128): number of rows:1285 are sent by batch! spent 54(ms)! 2013-05-24 22:00:42,905 DEBUG [WriterThread-1] wal.WALEditsReplaySink(128): number of rows:700 are sent by batch! spent 7(ms)! 2013-05-24 22:00:42,905 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,42690,1369432806911] wal.HLogSplitter$OutputSink(1276): Split writers finished 2013-05-24 22:00:42,906 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,42690,1369432806911] client.HConnectionManager$HConnectionImplementation(1640): Closing zookeeper sessionid=0x13ed88e2cdf0023 2013-05-24 22:00:42,913 DEBUG [WriterThread-2] wal.WALEditsReplaySink(128): number of rows:703 are sent by batch! spent 75(ms)! 2013-05-24 22:00:42,940 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392.logRoller] regionserver.LogRoller(90): HLog roll requested 2013-05-24 22:00:42,941 DEBUG [WriterThread-2] wal.WALEditsReplaySink(128): number of rows:1304 are sent by batch! spent 18(ms)! 2013-05-24 22:00:42,941 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] wal.HLogSplitter$OutputSink(1276): Split writers finished 2013-05-24 22:00:42,942 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] client.HConnectionManager$HConnectionImplementation(1640): Closing zookeeper sessionid=0x13ed88e2cdf0024 2013-05-24 22:00:42,955 INFO [Thread-848] regionserver.ReplicationSource$2(804): Slave cluster looks down: This server is in the failed servers list: kiyo.gq1.ygridcore.net/68.142.244.61:39146 2013-05-24 22:00:42,967 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,42690,1369432806911] wal.HLogSplitter(602): Processed 705 edits across 3 regions; log file=hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997-splitting/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432832342 is corrupted = false progress failed = false 2013-05-24 22:00:43,028 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.logRoller] wal.FSHLog(658): cleanupCurrentWriter waiting for transactions to get synced total 3695 synced till here 3694 2013-05-24 22:00:43,029 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] wal.HLogSplitter(602): Processed 4 edits across 1 regions; log file=hdfs://localhost:54104/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,39146,1369432814418-splitting/kiyo.gq1.ygridcore.net%2C39146%2C1369432814418.1369432818197 is corrupted = false progress failed = false 2013-05-24 22:00:43,031 INFO [IPC Server handler 3 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59993 is added to blk_4773416894237475455_1045{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:53333|RBW], ReplicaUnderConstruction[127.0.0.1:59993|RBW]]} size 0 2013-05-24 22:00:43,032 INFO [IPC Server handler 9 on 54104] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33054 is added to blk_1304456823603837391_1018{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:52910|RBW], ReplicaUnderConstruction[127.0.0.1:33054|RBW]]} size 0 2013-05-24 22:00:43,032 INFO [IPC Server handler 1 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53333 is added to blk_4773416894237475455_1045 size 84693 2013-05-24 22:00:43,032 INFO [IPC Server handler 8 on 54104] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52910 is added to blk_1304456823603837391_1018 size 46303 2013-05-24 22:00:43,055 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(583): Since we are unable to replicate, sleeping 100 times 2 2013-05-24 22:00:43,077 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:42690-0x13ed88e2cdf0006 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C49701%2C1369432806997-splitting%2Fkiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432832342 2013-05-24 22:00:43,077 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C49701%2C1369432806997-splitting%2Fkiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432832342 2013-05-24 22:00:43,077 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,42690,1369432806911] regionserver.SplitLogWorker(451): successfully transitioned task /1/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C49701%2C1369432806997-splitting%2Fkiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432832342 to final state DONE kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:43,078 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,42690,1369432806911] regionserver.SplitLogWorker(385): worker kiyo.gq1.ygridcore.net,42690,1369432806911 done with task /1/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C49701%2C1369432806997-splitting%2Fkiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432832342 in 446ms 2013-05-24 22:00:43,151 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392.logRoller] wal.FSHLog(522): Rolled WAL /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,51523,1369432814392/kiyo.gq1.ygridcore.net%2C51523%2C1369432814392.1369432841809, entries=2, filesize=45.2 K; new WAL=/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,51523,1369432814392/kiyo.gq1.ygridcore.net%2C51523%2C1369432814392.1369432842941 2013-05-24 22:00:43,184 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.logRoller] wal.FSHLog(522): Rolled WAL /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432841823, entries=371, filesize=82.7 K; new WAL=/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432842898 2013-05-24 22:00:43,204 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:51523-0x13ed88e2cdf0010 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432818197 2013-05-24 22:00:43,204 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:53632-0x13ed88e2cdf000f Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432818197 2013-05-24 22:00:43,204 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(716): task /1/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C49701%2C1369432806997-splitting%2Fkiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432832342 entered state: DONE kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:00:43,205 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] regionserver.SplitLogWorker(451): successfully transitioned task /2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432818197 to final state DONE kiyo.gq1.ygridcore.net,51523,1369432814392 2013-05-24 22:00:43,205 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] regionserver.SplitLogWorker(385): worker kiyo.gq1.ygridcore.net,51523,1369432814392 done with task /2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432818197 in 634ms 2013-05-24 22:00:43,205 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(716): task /2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432818197 entered state: DONE kiyo.gq1.ygridcore.net,51523,1369432814392 2013-05-24 22:00:43,242 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] regionserver.SplitLogWorker(330): worker kiyo.gq1.ygridcore.net,51523,1369432814392 acquired task /2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432833356 2013-05-24 22:00:43,244 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] wal.HLogSplitter(506): Splitting hlog: hdfs://localhost:54104/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,39146,1369432814418-splitting/kiyo.gq1.ygridcore.net%2C39146%2C1369432814418.1369432833356, length=37779 2013-05-24 22:00:43,244 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] wal.HLogSplitter(507): DistributedLogReplay = true 2013-05-24 22:00:43,256 INFO [Thread-857] regionserver.ReplicationSource$2(804): Slave cluster looks down: This server is in the failed servers list: kiyo.gq1.ygridcore.net/68.142.244.61:39146 2013-05-24 22:00:43,257 DEBUG [kiyo.gq1.ygridcore.net,49101,1369432806564.splitLogManagerTimeoutMonitor] master.SplitLogManager$TimeoutMonitor(1337): total tasks = 1 unassigned = 0 2013-05-24 22:00:43,260 DEBUG [pool-1-thread-1-EventThread] wal.HLogSplitter(692): Archived processed log hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997-splitting/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432832342 to hdfs://localhost:54096/user/hortonzy/hbase/.oldlogs/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432832342 2013-05-24 22:00:43,261 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(629): Done splitting /1/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C49701%2C1369432806997-splitting%2Fkiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432832342 2013-05-24 22:00:43,276 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:51523-0x13ed88e2cdf0010 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432833356 2013-05-24 22:00:43,276 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] util.FSHDFSUtils(86): Recovering lease on dfs file hdfs://localhost:54104/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,39146,1369432814418-splitting/kiyo.gq1.ygridcore.net%2C39146%2C1369432814418.1369432833356 2013-05-24 22:00:43,277 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] util.FSHDFSUtils(156): recoverLease=true, attempt=0 on file=hdfs://localhost:54104/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,39146,1369432814418-splitting/kiyo.gq1.ygridcore.net%2C39146%2C1369432814418.1369432833356 after 1ms 2013-05-24 22:00:43,281 DEBUG [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 37771, fileLength: 37779, trailerPresent: true 2013-05-24 22:00:43,300 DEBUG [pool-1-thread-1-EventThread] wal.HLogSplitter(692): Archived processed log hdfs://localhost:54104/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,39146,1369432814418-splitting/kiyo.gq1.ygridcore.net%2C39146%2C1369432814418.1369432818197 to hdfs://localhost:54104/user/hortonzy/hbase/.oldlogs/kiyo.gq1.ygridcore.net%2C39146%2C1369432814418.1369432818197 2013-05-24 22:00:43,301 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(629): Done splitting /2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432818197 2013-05-24 22:00:43,302 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:53632-0x13ed88e2cdf000f Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432833356 2013-05-24 22:00:43,307 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C49701%2C1369432806997-splitting%2Fkiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432832342 2013-05-24 22:00:43,307 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:42690-0x13ed88e2cdf0006 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/splitlog 2013-05-24 22:00:43,308 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] regionserver.SplitLogWorker(572): tasks arrived or departed 2013-05-24 22:00:43,310 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager$DeleteAsyncCallback(1529): deleted /1/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C49701%2C1369432806997-splitting%2Fkiyo.gq1.ygridcore.net%252C49701%252C1369432806997.1369432832342 2013-05-24 22:00:43,315 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:53632-0x13ed88e2cdf000f Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432818197 2013-05-24 22:00:43,316 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager$DeleteAsyncCallback(1529): deleted /2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432818197 2013-05-24 22:00:43,318 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(781): task /2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432833356 acquired by kiyo.gq1.ygridcore.net,51523,1369432814392 2013-05-24 22:00:43,320 DEBUG [WriterThread-0] wal.HLogSplitter$WriterThread(1102): Writer thread Thread[WriterThread-0,5,main]: starting 2013-05-24 22:00:43,320 DEBUG [WriterThread-1] wal.HLogSplitter$WriterThread(1102): Writer thread Thread[WriterThread-1,5,main]: starting 2013-05-24 22:00:43,320 DEBUG [WriterThread-2] wal.HLogSplitter$WriterThread(1102): Writer thread Thread[WriterThread-2,5,main]: starting 2013-05-24 22:00:43,326 INFO [WriterThread-2] zookeeper.RecoverableZooKeeper(119): Process identifier=hconnection-0x38a06569 connecting to ZooKeeper ensemble=localhost:51411 2013-05-24 22:00:43,327 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] wal.HLogSplitter(595): Finishing writing output logs and closing down. 2013-05-24 22:00:43,328 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] wal.HLogSplitter$OutputSink(1258): Waiting for split writer threads to finish 2013-05-24 22:00:43,342 INFO [kiyo.gq1.ygridcore.net,39146,1369432814418Replication Statistics #0] regionserver.Replication$ReplicationStatisticsThread(291): Sink: age in ms of last applied edit: 8247, total replicated edits: 2115 2013-05-24 22:00:43,355 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(583): Since we are unable to replicate, sleeping 100 times 3 2013-05-24 22:00:43,375 DEBUG [WriterThread-2-EventThread] zookeeper.ZooKeeperWatcher(307): hconnection-0x38a06569 Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2013-05-24 22:00:43,376 DEBUG [WriterThread-2-EventThread] zookeeper.ZooKeeperWatcher(384): hconnection-0x38a06569-0x13ed88e2cdf0025 connected 2013-05-24 22:00:43,406 INFO [kiyo.gq1.ygridcore.net,51523,1369432814392Replication Statistics #0] regionserver.Replication$ReplicationStatisticsThread(291): Sink: age in ms of last applied edit: 9804, total replicated edits: 1629 2013-05-24 22:00:43,410 DEBUG [ReplicationExecutor-0.replicationSource,2-kiyo.gq1.ygridcore.net,49701,1369432806997] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 68784, fileLength: 68792, trailerPresent: true 2013-05-24 22:00:43,418 INFO [Thread-584] replication.TestReplicationQueueFailover(63): Done loading table 2013-05-24 22:00:43,418 INFO [Thread-584] replication.TestReplicationQueueFailover(66): Done waiting for threads 2013-05-24 22:00:43,418 DEBUG [Thread-584] client.ClientScanner(96): Creating scanner over test starting at key '' 2013-05-24 22:00:43,418 DEBUG [Thread-584] client.ClientScanner(209): Advancing internal scanner to startKey at '' 2013-05-24 22:00:43,419 WARN [Thread-584] client.ServerCallable(172): Call exception, tries=0, numRetries=10: java.net.ConnectException: Connection refused 2013-05-24 22:00:43,419 DEBUG [Thread-584] client.HConnectionManager$HConnectionImplementation(1113): Removed all cached region locations that map to kiyo.gq1.ygridcore.net,49701,1369432806997 2013-05-24 22:00:43,423 DEBUG [ReplicationExecutor-0.replicationSource,2-kiyo.gq1.ygridcore.net,49701,1369432806997] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 46226, fileLength: 46234, trailerPresent: true 2013-05-24 22:00:43,499 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:42690-0x13ed88e2cdf0006 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/recovering-regions/0d6df81a73ad3b2a79add80716da62ac 2013-05-24 22:00:43,499 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.RecoveringRegionWatcher(66): /1/recovering-regions/0d6df81a73ad3b2a79add80716da62ac znode deleted. Region: 0d6df81a73ad3b2a79add80716da62ac completes recovery. 2013-05-24 22:00:43,501 INFO [REPL IPC Server handler 0 on 51523] zookeeper.RecoverableZooKeeper(119): Process identifier=hconnection-0x32a3d197 connecting to ZooKeeper ensemble=localhost:51411 2013-05-24 22:00:43,527 DEBUG [REPL IPC Server handler 0 on 51523-EventThread] zookeeper.ZooKeeperWatcher(307): hconnection-0x32a3d197 Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2013-05-24 22:00:43,529 DEBUG [REPL IPC Server handler 0 on 51523-EventThread] zookeeper.ZooKeeperWatcher(384): hconnection-0x32a3d197-0x13ed88e2cdf0026 connected 2013-05-24 22:00:43,634 DEBUG [kiyo.gq1.ygridcore.net,53632,1369432814258.splitLogManagerTimeoutMonitor] master.SplitLogManager$TimeoutMonitor(1337): total tasks = 1 unassigned = 0 2013-05-24 22:00:43,637 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:42690-0x13ed88e2cdf0006 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/recovering-regions/eb38c552d2a930aa1ce862202da80908 2013-05-24 22:00:43,637 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.RecoveringRegionWatcher(66): /1/recovering-regions/eb38c552d2a930aa1ce862202da80908 znode deleted. Region: eb38c552d2a930aa1ce862202da80908 completes recovery. 2013-05-24 22:00:43,642 DEBUG [REPL IPC Server handler 0 on 51523] client.MetaScanner(160): Scanning .META. starting at row=test,,00000000000000 for max=10 with caching=10 2013-05-24 22:00:43,642 DEBUG [WriterThread-2] client.MetaScanner(160): Scanning .META. starting at row=test,,00000000000000 for max=10 with caching=10 2013-05-24 22:00:43,642 DEBUG [REPL IPC Server handler 0 on 51523] client.ClientScanner(96): Creating scanner over .META. starting at key 'test,,00000000000000' 2013-05-24 22:00:43,642 DEBUG [WriterThread-2] client.ClientScanner(96): Creating scanner over .META. starting at key 'test,,00000000000000' 2013-05-24 22:00:43,642 DEBUG [REPL IPC Server handler 0 on 51523] client.ClientScanner(209): Advancing internal scanner to startKey at 'test,,00000000000000' 2013-05-24 22:00:43,643 DEBUG [WriterThread-2] client.ClientScanner(209): Advancing internal scanner to startKey at 'test,,00000000000000' 2013-05-24 22:00:43,648 DEBUG [WriterThread-2] client.ClientScanner(196): Finished scanning region {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,} 2013-05-24 22:00:43,648 DEBUG [REPL IPC Server handler 0 on 51523] client.ClientScanner(196): Finished scanning region {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,} 2013-05-24 22:00:43,648 DEBUG [WriterThread-2] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,,1369432825000.45883ac87958f956040ff389c9f7eb86. is kiyo.gq1.ygridcore.net:51523 2013-05-24 22:00:43,648 DEBUG [REPL IPC Server handler 0 on 51523] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,,1369432825000.45883ac87958f956040ff389c9f7eb86. is kiyo.gq1.ygridcore.net:51523 2013-05-24 22:00:43,649 INFO [REPL IPC Server handler 0 on 51523] client.HConnectionManager$HConnectionImplementation(1640): Closing zookeeper sessionid=0x13ed88e2cdf0026 2013-05-24 22:00:43,656 INFO [Thread-864] regionserver.ReplicationSource$2(804): Slave cluster looks down: This server is in the failed servers list: kiyo.gq1.ygridcore.net/68.142.244.61:39146 2013-05-24 22:00:43,756 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(583): Since we are unable to replicate, sleeping 100 times 4 2013-05-24 22:00:43,807 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:42690-0x13ed88e2cdf0006 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/recovering-regions/dba8b67ade91438cad8a5209736330e6 2013-05-24 22:00:43,807 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.RecoveringRegionWatcher(66): /1/recovering-regions/dba8b67ade91438cad8a5209736330e6 znode deleted. Region: dba8b67ade91438cad8a5209736330e6 completes recovery. 2013-05-24 22:00:43,821 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392.logRoller] regionserver.LogRoller(90): HLog roll requested 2013-05-24 22:00:43,860 DEBUG [WriterThread-2] wal.WALEditsReplaySink(128): number of rows:1622 are sent by batch! spent 65(ms)! 2013-05-24 22:00:43,860 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] wal.HLogSplitter$OutputSink(1276): Split writers finished 2013-05-24 22:00:43,861 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] client.HConnectionManager$HConnectionImplementation(1640): Closing zookeeper sessionid=0x13ed88e2cdf0025 2013-05-24 22:00:43,875 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:42690-0x13ed88e2cdf0006 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/recovering-regions/4b3ff3efde12c5afe1220e81d155ac73 2013-05-24 22:00:43,875 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.RecoveringRegionWatcher(66): /1/recovering-regions/4b3ff3efde12c5afe1220e81d155ac73 znode deleted. Region: 4b3ff3efde12c5afe1220e81d155ac73 completes recovery. 2013-05-24 22:00:43,894 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:42690-0x13ed88e2cdf0006 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/replication/rs/kiyo.gq1.ygridcore.net,42690,1369432806911/2-kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432829856 2013-05-24 22:00:43,895 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] wal.HLogSplitter(602): Processed 8 edits across 1 regions; log file=hdfs://localhost:54104/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,39146,1369432814418-splitting/kiyo.gq1.ygridcore.net%2C39146%2C1369432814418.1369432833356 is corrupted = false progress failed = false 2013-05-24 22:00:43,900 DEBUG [ReplicationExecutor-0.replicationSource,2-kiyo.gq1.ygridcore.net,49701,1369432806997] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 46226, fileLength: 46234, trailerPresent: true 2013-05-24 22:00:43,907 INFO [IPC Server handler 8 on 54104] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33054 is added to blk_-7898610662665388472_1019{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:52910|RBW], ReplicaUnderConstruction[127.0.0.1:33054|RBW]]} size 0 2013-05-24 22:00:43,908 INFO [IPC Server handler 0 on 54104] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52910 is added to blk_-7898610662665388472_1019 size 69886 2013-05-24 22:00:43,912 DEBUG [ReplicationExecutor-0.replicationSource,2-kiyo.gq1.ygridcore.net,49701,1369432806997] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 69060, fileLength: 69068, trailerPresent: true 2013-05-24 22:00:43,931 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:51523-0x13ed88e2cdf0010 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432833356 2013-05-24 22:00:43,932 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:53632-0x13ed88e2cdf000f Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432833356 2013-05-24 22:00:43,933 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] regionserver.SplitLogWorker(451): successfully transitioned task /2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432833356 to final state DONE kiyo.gq1.ygridcore.net,51523,1369432814392 2013-05-24 22:00:43,933 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] regionserver.SplitLogWorker(385): worker kiyo.gq1.ygridcore.net,51523,1369432814392 done with task /2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432833356 in 691ms 2013-05-24 22:00:43,935 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(716): task /2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432833356 entered state: DONE kiyo.gq1.ygridcore.net,51523,1369432814392 2013-05-24 22:00:44,011 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392.logRoller] wal.FSHLog(522): Rolled WAL /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,51523,1369432814392/kiyo.gq1.ygridcore.net%2C51523%2C1369432814392.1369432842941, entries=2, filesize=68.2 K; new WAL=/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,51523,1369432814392/kiyo.gq1.ygridcore.net%2C51523%2C1369432814392.1369432843822 2013-05-24 22:00:44,096 DEBUG [pool-1-thread-1-EventThread] wal.HLogSplitter(692): Archived processed log hdfs://localhost:54104/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,39146,1369432814418-splitting/kiyo.gq1.ygridcore.net%2C39146%2C1369432814418.1369432833356 to hdfs://localhost:54104/user/hortonzy/hbase/.oldlogs/kiyo.gq1.ygridcore.net%2C39146%2C1369432814418.1369432833356 2013-05-24 22:00:44,097 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(629): Done splitting /2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432833356 2013-05-24 22:00:44,157 INFO [Thread-878] regionserver.ReplicationSource$2(804): Slave cluster looks down: This server is in the failed servers list: kiyo.gq1.ygridcore.net/68.142.244.61:39146 2013-05-24 22:00:44,183 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392.logRoller] regionserver.LogRoller(90): HLog roll requested 2013-05-24 22:00:44,208 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:42690-0x13ed88e2cdf0006 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/recovering-regions/a88e8c9e73f832adb3635d6996d9e72d 2013-05-24 22:00:44,208 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.RecoveringRegionWatcher(66): /1/recovering-regions/a88e8c9e73f832adb3635d6996d9e72d znode deleted. Region: a88e8c9e73f832adb3635d6996d9e72d completes recovery. 2013-05-24 22:00:44,257 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(583): Since we are unable to replicate, sleeping 100 times 5 2013-05-24 22:00:44,328 INFO [IPC Server handler 9 on 54104] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52910 is added to blk_-1256644577614532178_1020{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:33054|RBW], ReplicaUnderConstruction[127.0.0.1:52910|RBW]]} size 0 2013-05-24 22:00:44,328 INFO [IPC Server handler 0 on 54104] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33054 is added to blk_-1256644577614532178_1020 size 48612 2013-05-24 22:00:44,351 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:42690-0x13ed88e2cdf0006 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/replication/rs/kiyo.gq1.ygridcore.net,42690,1369432806911/2-kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432832342 2013-05-24 22:00:44,355 DEBUG [ReplicationExecutor-0.replicationSource,2-kiyo.gq1.ygridcore.net,49701,1369432806997] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 69060, fileLength: 69068, trailerPresent: true 2013-05-24 22:00:44,366 DEBUG [ReplicationExecutor-0.replicationSource,2-kiyo.gq1.ygridcore.net,49701,1369432806997] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 23028, fileLength: 23036, trailerPresent: true 2013-05-24 22:00:44,399 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:53632-0x13ed88e2cdf000f Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432833356 2013-05-24 22:00:44,399 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:51523-0x13ed88e2cdf0010 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/2/splitlog 2013-05-24 22:00:44,399 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392-EventThread] regionserver.SplitLogWorker(572): tasks arrived or departed 2013-05-24 22:00:44,401 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager$DeleteAsyncCallback(1529): deleted /2/splitlog/.logs%2Fkiyo.gq1.ygridcore.net%2C39146%2C1369432814418-splitting%2Fkiyo.gq1.ygridcore.net%252C39146%252C1369432814418.1369432833356 2013-05-24 22:00:44,415 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392.logRoller] wal.FSHLog(522): Rolled WAL /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,51523,1369432814392/kiyo.gq1.ygridcore.net%2C51523%2C1369432814392.1369432843822, entries=1, filesize=47.5 K; new WAL=/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,51523,1369432814392/kiyo.gq1.ygridcore.net%2C51523%2C1369432814392.1369432844184 2013-05-24 22:00:44,459 DEBUG [Thread-584] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,,1369432820315.a88e8c9e73f832adb3635d6996d9e72d. is kiyo.gq1.ygridcore.net:42690 2013-05-24 22:00:44,485 DEBUG [Thread-584] client.ClientScanner(202): Finished with region {NAME => 'test,,1369432820315.a88e8c9e73f832adb3635d6996d9e72d.', STARTKEY => '', ENDKEY => 'bbb', ENCODED => a88e8c9e73f832adb3635d6996d9e72d,} 2013-05-24 22:00:44,485 DEBUG [Thread-584] client.ClientScanner(209): Advancing internal scanner to startKey at 'bbb' 2013-05-24 22:00:44,513 DEBUG [Thread-584] client.ClientScanner(202): Finished with region {NAME => 'test,bbb,1369432820319.6a4e618fc444a41a507e36bb35c860a6.', STARTKEY => 'bbb', ENDKEY => 'ccc', ENCODED => 6a4e618fc444a41a507e36bb35c860a6,} 2013-05-24 22:00:44,513 DEBUG [Thread-584] client.ClientScanner(209): Advancing internal scanner to startKey at 'ccc' 2013-05-24 22:00:44,518 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:42690-0x13ed88e2cdf0006 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/recovering-regions/ec0b9e20bb8d4dc8fee9b3e80c5cae32 2013-05-24 22:00:44,519 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.RecoveringRegionWatcher(66): /1/recovering-regions/ec0b9e20bb8d4dc8fee9b3e80c5cae32 znode deleted. Region: ec0b9e20bb8d4dc8fee9b3e80c5cae32 completes recovery. 2013-05-24 22:00:44,537 DEBUG [Thread-584] client.ClientScanner(202): Finished with region {NAME => 'test,ccc,1369432820319.117d9d6b052965f7196396ea21f07a61.', STARTKEY => 'ccc', ENDKEY => 'ddd', ENCODED => 117d9d6b052965f7196396ea21f07a61,} 2013-05-24 22:00:44,537 DEBUG [Thread-584] client.ClientScanner(209): Advancing internal scanner to startKey at 'ddd' 2013-05-24 22:00:44,548 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:42690-0x13ed88e2cdf0006 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/replication/rs/kiyo.gq1.ygridcore.net,42690,1369432806911/2-kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432833549 2013-05-24 22:00:44,553 DEBUG [ReplicationExecutor-0.replicationSource,2-kiyo.gq1.ygridcore.net,49701,1369432806997] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 23028, fileLength: 23036, trailerPresent: true 2013-05-24 22:00:44,558 INFO [ReplicationExecutor-0.replicationSource,2-kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.ReplicationSourceManager(366): Done with the recovered queue 2-kiyo.gq1.ygridcore.net,49701,1369432806997 2013-05-24 22:00:44,558 DEBUG [Thread-584] client.ClientScanner(202): Finished with region {NAME => 'test,ddd,1369432820319.925080806d078451834e3417849dd4a9.', STARTKEY => 'ddd', ENDKEY => 'eee', ENCODED => 925080806d078451834e3417849dd4a9,} 2013-05-24 22:00:44,558 DEBUG [Thread-584] client.ClientScanner(209): Advancing internal scanner to startKey at 'eee' 2013-05-24 22:00:44,585 DEBUG [Thread-584] client.ClientScanner(202): Finished with region {NAME => 'test,eee,1369432820319.6f0a98b71d64131085456015699d44f0.', STARTKEY => 'eee', ENDKEY => 'fff', ENCODED => 6f0a98b71d64131085456015699d44f0,} 2013-05-24 22:00:44,585 DEBUG [Thread-584] client.ClientScanner(209): Advancing internal scanner to startKey at 'fff' 2013-05-24 22:00:44,631 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:51523-0x13ed88e2cdf0010 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/2/recovering-regions/45883ac87958f956040ff389c9f7eb86 2013-05-24 22:00:44,632 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392-EventThread] zookeeper.RecoveringRegionWatcher(66): /2/recovering-regions/45883ac87958f956040ff389c9f7eb86 znode deleted. Region: 45883ac87958f956040ff389c9f7eb86 completes recovery. 2013-05-24 22:00:44,703 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,53632,1369432814258-0] master.SplitLogManager(357): finished splitting (more than or equal to) 261237 bytes in 6 log files in [hdfs://localhost:54104/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,39146,1369432814418-splitting] in 5633ms 2013-05-24 22:00:44,703 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,53632,1369432814258-0] handler.ServerShutdownHandler(304): Finished processing of shutdown of kiyo.gq1.ygridcore.net,39146,1369432814418 2013-05-24 22:00:44,758 INFO [Thread-886] regionserver.ReplicationSource$2(804): Slave cluster looks down: Connection refused 2013-05-24 22:00:44,768 DEBUG [Thread-584] client.MetaScanner(160): Scanning .META. starting at row=test,fff,00000000000000 for max=10 with caching=10 2013-05-24 22:00:44,768 DEBUG [Thread-584] client.ClientScanner(96): Creating scanner over .META. starting at key 'test,fff,00000000000000' 2013-05-24 22:00:44,768 DEBUG [Thread-584] client.ClientScanner(209): Advancing internal scanner to startKey at 'test,fff,00000000000000' 2013-05-24 22:00:44,791 INFO [kiyo.gq1.ygridcore.net,49701,1369432806997Replication Statistics #0] regionserver.Replication$ReplicationStatisticsThread(291): Normal source for cluster 2: Total replicated edits: 1629, currently replicating from: hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432832342 at position: N/A 2013-05-24 22:00:44,791 INFO [kiyo.gq1.ygridcore.net,42690,1369432806911Replication Statistics #0] regionserver.Replication$ReplicationStatisticsThread(291): Normal source for cluster 2: Total replicated edits: 2115, currently replicating from: hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432839980 at position: N/A 2013-05-24 22:00:44,825 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:42690-0x13ed88e2cdf0006 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/recovering-regions/acf78311e603cb0d02309ae08763d196 2013-05-24 22:00:44,825 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.RecoveringRegionWatcher(66): /1/recovering-regions/acf78311e603cb0d02309ae08763d196 znode deleted. Region: acf78311e603cb0d02309ae08763d196 completes recovery. 2013-05-24 22:00:44,858 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(583): Since we are unable to replicate, sleeping 100 times 6 2013-05-24 22:00:44,873 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:42690-0x13ed88e2cdf0006 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/replication/rs/kiyo.gq1.ygridcore.net,42690,1369432806911/2-kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432809419 2013-05-24 22:00:44,877 DEBUG [Thread-584] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,fff,1369432820319.f852f52d87b0b7acd213f5b49074dde6. is kiyo.gq1.ygridcore.net:42690 2013-05-24 22:00:44,878 DEBUG [Thread-584] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,ggg,1369432820319.4b3ff3efde12c5afe1220e81d155ac73. is kiyo.gq1.ygridcore.net:42690 2013-05-24 22:00:44,878 DEBUG [Thread-584] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,jjj,1369432820320.eb38c552d2a930aa1ce862202da80908. is kiyo.gq1.ygridcore.net:42690 2013-05-24 22:00:44,878 DEBUG [Thread-584] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,kkk,1369432820320.1ce9aafaaa422cef4ce0e1895111c733. is kiyo.gq1.ygridcore.net:42690 2013-05-24 22:00:44,879 DEBUG [Thread-584] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,mmm,1369432820320.ab7b9b9773243cff0ac09f3c379e33dc. is kiyo.gq1.ygridcore.net:42690 2013-05-24 22:00:44,879 DEBUG [Thread-584] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,nnn,1369432820320.53e3aef4cc8815dd31923659bff5ca21. is kiyo.gq1.ygridcore.net:42690 2013-05-24 22:00:44,879 DEBUG [Thread-584] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,ooo,1369432820320.0d6df81a73ad3b2a79add80716da62ac. is kiyo.gq1.ygridcore.net:42690 2013-05-24 22:00:44,880 DEBUG [IPC Server handler 2 on 42690] regionserver.HRegionServer(2562): NotServingRegionException; test,fff,1369432820319.f852f52d87b0b7acd213f5b49074dde6. is recovering 2013-05-24 22:00:44,882 WARN [Thread-584] client.ServerCallable(172): Call exception, tries=0, numRetries=10: org.apache.hadoop.hbase.exceptions.RegionInRecoveryException: org.apache.hadoop.hbase.exceptions.RegionInRecoveryException: test,fff,1369432820319.f852f52d87b0b7acd213f5b49074dde6. is recovering at org.apache.hadoop.hbase.regionserver.HRegion.startRegionOperation(HRegion.java:5336) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1687) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1673) at org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:2983) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:20800) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2103) at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1810) 2013-05-24 22:00:44,932 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:42690-0x13ed88e2cdf0006 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/recovering-regions/f852f52d87b0b7acd213f5b49074dde6 2013-05-24 22:00:44,932 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.RecoveringRegionWatcher(66): /1/recovering-regions/f852f52d87b0b7acd213f5b49074dde6 znode deleted. Region: f852f52d87b0b7acd213f5b49074dde6 completes recovery. 2013-05-24 22:00:44,985 INFO [ReplicationExecutor-0.replicationSource,2-kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.ReplicationSource(734): Finished recovering the queue 2013-05-24 22:00:44,985 DEBUG [ReplicationExecutor-0.replicationSource,2-kiyo.gq1.ygridcore.net,49701,1369432806997] regionserver.ReplicationSource(388): Source exiting 2 2013-05-24 22:00:45,048 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:42690-0x13ed88e2cdf0006 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/recovering-regions/ab7b9b9773243cff0ac09f3c379e33dc 2013-05-24 22:00:45,048 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.RecoveringRegionWatcher(66): /1/recovering-regions/ab7b9b9773243cff0ac09f3c379e33dc znode deleted. Region: ab7b9b9773243cff0ac09f3c379e33dc completes recovery. 2013-05-24 22:00:45,235 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:42690-0x13ed88e2cdf0006 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/recovering-regions/1ce9aafaaa422cef4ce0e1895111c733 2013-05-24 22:00:45,235 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.RecoveringRegionWatcher(66): /1/recovering-regions/1ce9aafaaa422cef4ce0e1895111c733 znode deleted. Region: 1ce9aafaaa422cef4ce0e1895111c733 completes recovery. 2013-05-24 22:00:45,442 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:42690-0x13ed88e2cdf0006 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/recovering-regions/0b4d70d096cb6cbce3376acd563c77dd 2013-05-24 22:00:45,442 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.RecoveringRegionWatcher(66): /1/recovering-regions/0b4d70d096cb6cbce3376acd563c77dd znode deleted. Region: 0b4d70d096cb6cbce3376acd563c77dd completes recovery. 2013-05-24 22:00:45,459 INFO [Thread-888] regionserver.ReplicationSource$2(804): Slave cluster looks down: This server is in the failed servers list: kiyo.gq1.ygridcore.net/68.142.244.61:39146 2013-05-24 22:00:45,536 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:42690-0x13ed88e2cdf0006 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/recovering-regions/53e3aef4cc8815dd31923659bff5ca21 2013-05-24 22:00:45,536 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.RecoveringRegionWatcher(66): /1/recovering-regions/53e3aef4cc8815dd31923659bff5ca21 znode deleted. Region: 53e3aef4cc8815dd31923659bff5ca21 completes recovery. 2013-05-24 22:00:45,558 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(583): Since we are unable to replicate, sleeping 100 times 7 2013-05-24 22:00:45,974 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:42690-0x13ed88e2cdf0006 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/recovering-regions/872960a61a4ceeb9f64577d8b7cb0ddf 2013-05-24 22:00:45,975 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.RecoveringRegionWatcher(66): /1/recovering-regions/872960a61a4ceeb9f64577d8b7cb0ddf znode deleted. Region: 872960a61a4ceeb9f64577d8b7cb0ddf completes recovery. 2013-05-24 22:00:45,976 DEBUG [Thread-584] client.HConnectionManager$HConnectionImplementation(1085): Removed kiyo.gq1.ygridcore.net:42690 as a location of test,fff,1369432820319.f852f52d87b0b7acd213f5b49074dde6. for tableName=test from cache 2013-05-24 22:00:45,978 DEBUG [Thread-584] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,fff,1369432820319.f852f52d87b0b7acd213f5b49074dde6. is kiyo.gq1.ygridcore.net:42690 2013-05-24 22:00:46,002 DEBUG [Thread-584] client.ClientScanner(202): Finished with region {NAME => 'test,fff,1369432820319.f852f52d87b0b7acd213f5b49074dde6.', STARTKEY => 'fff', ENDKEY => 'ggg', ENCODED => f852f52d87b0b7acd213f5b49074dde6,} 2013-05-24 22:00:46,002 DEBUG [Thread-584] client.ClientScanner(209): Advancing internal scanner to startKey at 'ggg' 2013-05-24 22:00:46,017 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] master.SplitLogManager(357): finished splitting (more than or equal to) 207130 bytes in 4 log files in [hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997-splitting] in 6369ms 2013-05-24 22:00:46,017 INFO [MASTER_SERVER_OPERATIONS-kiyo.gq1.ygridcore.net,49101,1369432806564-0] handler.ServerShutdownHandler(304): Finished processing of shutdown of kiyo.gq1.ygridcore.net,49701,1369432806997 2013-05-24 22:00:46,025 DEBUG [Thread-584] client.ClientScanner(202): Finished with region {NAME => 'test,ggg,1369432820319.4b3ff3efde12c5afe1220e81d155ac73.', STARTKEY => 'ggg', ENDKEY => 'hhh', ENCODED => 4b3ff3efde12c5afe1220e81d155ac73,} 2013-05-24 22:00:46,025 DEBUG [Thread-584] client.ClientScanner(209): Advancing internal scanner to startKey at 'hhh' 2013-05-24 22:00:46,048 DEBUG [Thread-584] client.ClientScanner(202): Finished with region {NAME => 'test,hhh,1369432820320.a9a8fa0d1fb6a035656bc264ec313d2b.', STARTKEY => 'hhh', ENDKEY => 'iii', ENCODED => a9a8fa0d1fb6a035656bc264ec313d2b,} 2013-05-24 22:00:46,048 DEBUG [Thread-584] client.ClientScanner(209): Advancing internal scanner to startKey at 'iii' 2013-05-24 22:00:46,072 DEBUG [Thread-584] client.ClientScanner(202): Finished with region {NAME => 'test,iii,1369432820320.992fe9e8d6a399ebc508d777668f74ad.', STARTKEY => 'iii', ENDKEY => 'jjj', ENCODED => 992fe9e8d6a399ebc508d777668f74ad,} 2013-05-24 22:00:46,072 DEBUG [Thread-584] client.ClientScanner(209): Advancing internal scanner to startKey at 'jjj' 2013-05-24 22:00:46,089 DEBUG [Thread-584] client.ClientScanner(202): Finished with region {NAME => 'test,jjj,1369432820320.eb38c552d2a930aa1ce862202da80908.', STARTKEY => 'jjj', ENDKEY => 'kkk', ENCODED => eb38c552d2a930aa1ce862202da80908,} 2013-05-24 22:00:46,089 DEBUG [Thread-584] client.ClientScanner(209): Advancing internal scanner to startKey at 'kkk' 2013-05-24 22:00:46,101 DEBUG [Thread-584] client.ClientScanner(202): Finished with region {NAME => 'test,kkk,1369432820320.1ce9aafaaa422cef4ce0e1895111c733.', STARTKEY => 'kkk', ENDKEY => 'lll', ENCODED => 1ce9aafaaa422cef4ce0e1895111c733,} 2013-05-24 22:00:46,101 DEBUG [Thread-584] client.ClientScanner(209): Advancing internal scanner to startKey at 'lll' 2013-05-24 22:00:46,116 DEBUG [Thread-584] client.ClientScanner(202): Finished with region {NAME => 'test,lll,1369432820320.307a0833b0558971bccf01de5c996ebc.', STARTKEY => 'lll', ENDKEY => 'mmm', ENCODED => 307a0833b0558971bccf01de5c996ebc,} 2013-05-24 22:00:46,117 DEBUG [Thread-584] client.ClientScanner(209): Advancing internal scanner to startKey at 'mmm' 2013-05-24 22:00:46,129 DEBUG [Thread-584] client.ClientScanner(202): Finished with region {NAME => 'test,mmm,1369432820320.ab7b9b9773243cff0ac09f3c379e33dc.', STARTKEY => 'mmm', ENDKEY => 'nnn', ENCODED => ab7b9b9773243cff0ac09f3c379e33dc,} 2013-05-24 22:00:46,129 DEBUG [Thread-584] client.ClientScanner(209): Advancing internal scanner to startKey at 'nnn' 2013-05-24 22:00:46,139 DEBUG [Thread-584] client.ClientScanner(202): Finished with region {NAME => 'test,nnn,1369432820320.53e3aef4cc8815dd31923659bff5ca21.', STARTKEY => 'nnn', ENDKEY => 'ooo', ENCODED => 53e3aef4cc8815dd31923659bff5ca21,} 2013-05-24 22:00:46,140 DEBUG [Thread-584] client.ClientScanner(209): Advancing internal scanner to startKey at 'ooo' 2013-05-24 22:00:46,149 DEBUG [Thread-584] client.ClientScanner(202): Finished with region {NAME => 'test,ooo,1369432820320.0d6df81a73ad3b2a79add80716da62ac.', STARTKEY => 'ooo', ENDKEY => 'ppp', ENCODED => 0d6df81a73ad3b2a79add80716da62ac,} 2013-05-24 22:00:46,149 DEBUG [Thread-584] client.ClientScanner(209): Advancing internal scanner to startKey at 'ppp' 2013-05-24 22:00:46,157 DEBUG [Thread-584] client.ClientScanner(202): Finished with region {NAME => 'test,ppp,1369432820320.da89687648e34409286fd02380093aed.', STARTKEY => 'ppp', ENDKEY => 'qqq', ENCODED => da89687648e34409286fd02380093aed,} 2013-05-24 22:00:46,157 DEBUG [Thread-584] client.ClientScanner(209): Advancing internal scanner to startKey at 'qqq' 2013-05-24 22:00:46,161 DEBUG [Thread-584] client.MetaScanner(160): Scanning .META. starting at row=test,qqq,00000000000000 for max=10 with caching=10 2013-05-24 22:00:46,161 DEBUG [Thread-584] client.ClientScanner(96): Creating scanner over .META. starting at key 'test,qqq,00000000000000' 2013-05-24 22:00:46,161 DEBUG [Thread-584] client.ClientScanner(209): Advancing internal scanner to startKey at 'test,qqq,00000000000000' 2013-05-24 22:00:46,164 DEBUG [Thread-584] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,qqq,1369432820320.ec0b9e20bb8d4dc8fee9b3e80c5cae32. is kiyo.gq1.ygridcore.net:42690 2013-05-24 22:00:46,174 DEBUG [Thread-584] client.ClientScanner(202): Finished with region {NAME => 'test,qqq,1369432820320.ec0b9e20bb8d4dc8fee9b3e80c5cae32.', STARTKEY => 'qqq', ENDKEY => 'rrr', ENCODED => ec0b9e20bb8d4dc8fee9b3e80c5cae32,} 2013-05-24 22:00:46,174 DEBUG [Thread-584] client.ClientScanner(209): Advancing internal scanner to startKey at 'rrr' 2013-05-24 22:00:46,185 DEBUG [Thread-584] client.ClientScanner(202): Finished with region {NAME => 'test,rrr,1369432820320.85815417b0e1742517fac3960fac86c1.', STARTKEY => 'rrr', ENDKEY => 'sss', ENCODED => 85815417b0e1742517fac3960fac86c1,} 2013-05-24 22:00:46,186 DEBUG [Thread-584] client.ClientScanner(209): Advancing internal scanner to startKey at 'sss' 2013-05-24 22:00:46,194 DEBUG [Thread-584] client.ClientScanner(202): Finished with region {NAME => 'test,sss,1369432820320.872960a61a4ceeb9f64577d8b7cb0ddf.', STARTKEY => 'sss', ENDKEY => 'ttt', ENCODED => 872960a61a4ceeb9f64577d8b7cb0ddf,} 2013-05-24 22:00:46,194 DEBUG [Thread-584] client.ClientScanner(209): Advancing internal scanner to startKey at 'ttt' 2013-05-24 22:00:46,201 DEBUG [Thread-584] client.ClientScanner(202): Finished with region {NAME => 'test,ttt,1369432820320.f53d942877419dd243e3fb7e46c2cf97.', STARTKEY => 'ttt', ENDKEY => 'uuu', ENCODED => f53d942877419dd243e3fb7e46c2cf97,} 2013-05-24 22:00:46,201 DEBUG [Thread-584] client.ClientScanner(209): Advancing internal scanner to startKey at 'uuu' 2013-05-24 22:00:46,210 DEBUG [Thread-584] client.ClientScanner(202): Finished with region {NAME => 'test,uuu,1369432820320.dba8b67ade91438cad8a5209736330e6.', STARTKEY => 'uuu', ENDKEY => 'vvv', ENCODED => dba8b67ade91438cad8a5209736330e6,} 2013-05-24 22:00:46,210 DEBUG [Thread-584] client.ClientScanner(209): Advancing internal scanner to startKey at 'vvv' 2013-05-24 22:00:46,218 DEBUG [Thread-584] client.ClientScanner(202): Finished with region {NAME => 'test,vvv,1369432820320.84d2bf1edb7e9df4777e80207fc739d2.', STARTKEY => 'vvv', ENDKEY => 'www', ENCODED => 84d2bf1edb7e9df4777e80207fc739d2,} 2013-05-24 22:00:46,218 DEBUG [Thread-584] client.ClientScanner(209): Advancing internal scanner to startKey at 'www' 2013-05-24 22:00:46,224 DEBUG [Thread-584] client.ClientScanner(202): Finished with region {NAME => 'test,www,1369432820320.643c76d94d70be44d35b5a12853bb852.', STARTKEY => 'www', ENDKEY => 'xxx', ENCODED => 643c76d94d70be44d35b5a12853bb852,} 2013-05-24 22:00:46,225 DEBUG [Thread-584] client.ClientScanner(209): Advancing internal scanner to startKey at 'xxx' 2013-05-24 22:00:46,231 DEBUG [Thread-584] client.ClientScanner(202): Finished with region {NAME => 'test,xxx,1369432820320.b03c7b8b6013fa49f346edeb3a7cad87.', STARTKEY => 'xxx', ENDKEY => 'yyy', ENCODED => b03c7b8b6013fa49f346edeb3a7cad87,} 2013-05-24 22:00:46,232 DEBUG [Thread-584] client.ClientScanner(209): Advancing internal scanner to startKey at 'yyy' 2013-05-24 22:00:46,238 DEBUG [Thread-584] client.ClientScanner(202): Finished with region {NAME => 'test,yyy,1369432820321.acf78311e603cb0d02309ae08763d196.', STARTKEY => 'yyy', ENDKEY => 'zzz', ENCODED => acf78311e603cb0d02309ae08763d196,} 2013-05-24 22:00:46,238 DEBUG [Thread-584] client.ClientScanner(209): Advancing internal scanner to startKey at 'zzz' 2013-05-24 22:00:46,239 DEBUG [Thread-584] client.ClientScanner(196): Finished scanning region {NAME => 'test,zzz,1369432820321.0b4d70d096cb6cbce3376acd563c77dd.', STARTKEY => 'zzz', ENDKEY => '', ENCODED => 0b4d70d096cb6cbce3376acd563c77dd,} 2013-05-24 22:00:46,240 DEBUG [Thread-584] client.ClientScanner(96): Creating scanner over test starting at key '' 2013-05-24 22:00:46,240 DEBUG [Thread-584] client.ClientScanner(209): Advancing internal scanner to startKey at '' 2013-05-24 22:00:46,240 WARN [Thread-584] client.ServerCallable(172): Call exception, tries=0, numRetries=6: java.net.ConnectException: Connection refused 2013-05-24 22:00:46,241 DEBUG [Thread-584] client.HConnectionManager$HConnectionImplementation(1113): Removed all cached region locations that map to kiyo.gq1.ygridcore.net,39146,1369432814418 2013-05-24 22:00:46,259 INFO [Thread-891] regionserver.ReplicationSource$2(804): Slave cluster looks down: This server is in the failed servers list: kiyo.gq1.ygridcore.net/68.142.244.61:39146 2013-05-24 22:00:46,359 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(583): Since we are unable to replicate, sleeping 100 times 8 2013-05-24 22:00:47,160 INFO [Thread-893] regionserver.ReplicationSource$2(804): Slave cluster looks down: Connection refused 2013-05-24 22:00:47,253 DEBUG [Thread-584] client.HConnectionManager$HConnectionImplementation(1191): Cached location for test,,1369432825000.45883ac87958f956040ff389c9f7eb86. is kiyo.gq1.ygridcore.net:51523 2013-05-24 22:00:47,260 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(583): Since we are unable to replicate, sleeping 100 times 9 2013-05-24 22:00:47,327 DEBUG [Thread-584] client.ClientScanner(196): Finished scanning region {NAME => 'test,,1369432825000.45883ac87958f956040ff389c9f7eb86.', STARTKEY => '', ENDKEY => '', ENCODED => 45883ac87958f956040ff389c9f7eb86,} 2013-05-24 22:00:47,327 INFO [Thread-584] replication.TestReplicationQueueFailover(108): Only got 12654 rows instead of 17576 current i=-1 2013-05-24 22:00:48,161 INFO [Thread-896] regionserver.ReplicationSource$2(804): Slave cluster looks down: This server is in the failed servers list: kiyo.gq1.ygridcore.net/68.142.244.61:39146 2013-05-24 22:00:48,260 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(583): Since we are unable to replicate, sleeping 100 times 10 2013-05-24 22:00:48,328 DEBUG [Thread-584] client.ClientScanner(96): Creating scanner over test starting at key '' 2013-05-24 22:00:48,328 DEBUG [Thread-584] client.ClientScanner(209): Advancing internal scanner to startKey at '' 2013-05-24 22:00:48,342 INFO [kiyo.gq1.ygridcore.net,39146,1369432814418Replication Statistics #0] regionserver.Replication$ReplicationStatisticsThread(291): Sink: age in ms of last applied edit: 13247, total replicated edits: 2115 2013-05-24 22:00:48,396 DEBUG [Thread-584] client.ClientScanner(196): Finished scanning region {NAME => 'test,,1369432825000.45883ac87958f956040ff389c9f7eb86.', STARTKEY => '', ENDKEY => '', ENCODED => 45883ac87958f956040ff389c9f7eb86,} 2013-05-24 22:00:48,396 INFO [Thread-584] replication.TestReplicationQueueFailover(108): Only got 12654 rows instead of 17576 current i=0 2013-05-24 22:00:48,399 INFO [kiyo.gq1.ygridcore.net,51523,1369432814392Replication Statistics #0] regionserver.Replication$ReplicationStatisticsThread(291): Sink: age in ms of last applied edit: 13694, total replicated edits: 3041 2013-05-24 22:00:49,259 INFO [kiyo.gq1.ygridcore.net,49101,1369432806564.splitLogManagerTimeoutMonitor] master.SplitLogManager(604): Garbage collecting all recovering regions. 2013-05-24 22:00:49,261 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(211): Getting 1 rs from peer cluster # 2 2013-05-24 22:00:49,262 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(219): Choosing peer kiyo.gq1.ygridcore.net,51523,1369432814392 2013-05-24 22:00:49,314 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432839980 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:49,317 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 61479, fileLength: 61487, trailerPresent: true 2013-05-24 22:00:49,326 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432840428 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:49,330 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 132235, fileLength: 132243, trailerPresent: true 2013-05-24 22:00:49,337 ERROR [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(236): Error while reading 691 WAL KVs; started reading at 53272 and read up to 65538 2013-05-24 22:00:49,337 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(442): Break on IOE: Error while reading 691 WAL KVs; started reading at 53272 and read up to 65538 2013-05-24 22:00:49,372 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392.logRoller] regionserver.LogRoller(90): HLog roll requested 2013-05-24 22:00:49,397 DEBUG [Thread-584] client.ClientScanner(96): Creating scanner over test starting at key '' 2013-05-24 22:00:49,397 DEBUG [Thread-584] client.ClientScanner(209): Advancing internal scanner to startKey at '' 2013-05-24 22:00:49,470 DEBUG [Thread-584] client.ClientScanner(196): Finished scanning region {NAME => 'test,,1369432825000.45883ac87958f956040ff389c9f7eb86.', STARTKEY => '', ENDKEY => '', ENCODED => 45883ac87958f956040ff389c9f7eb86,} 2013-05-24 22:00:49,471 INFO [Thread-584] replication.TestReplicationQueueFailover(108): Only got 14304 rows instead of 17576 current i=-1 2013-05-24 22:00:49,586 INFO [IPC Server handler 2 on 54104] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33054 is added to blk_3044260377110907568_1021{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:52910|RBW], ReplicaUnderConstruction[127.0.0.1:33054|RBW]]} size 0 2013-05-24 22:00:49,586 INFO [IPC Server handler 7 on 54104] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52910 is added to blk_3044260377110907568_1021 size 54330 2013-05-24 22:00:49,607 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432840428 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:49,611 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 132235, fileLength: 132243, trailerPresent: true 2013-05-24 22:00:49,612 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392.logRoller] wal.FSHLog(522): Rolled WAL /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,51523,1369432814392/kiyo.gq1.ygridcore.net%2C51523%2C1369432814392.1369432844184, entries=3, filesize=53.1 K; new WAL=/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,51523,1369432814392/kiyo.gq1.ygridcore.net%2C51523%2C1369432814392.1369432849372 2013-05-24 22:00:49,621 ERROR [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(236): Error while reading 691 WAL KVs; started reading at 53272 and read up to 65538 2013-05-24 22:00:49,621 WARN [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(324): 2 Got: java.io.IOException: Error while reading 691 WAL KVs; started reading at 53272 and read up to 65538 at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:237) at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:96) at org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.readNextAndSetPosition(ReplicationHLogReaderManager.java:89) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.readAllEntriesToReplicateOrNextFile(ReplicationSource.java:404) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:320) Caused by: java.lang.IndexOutOfBoundsException: index (30062) must be less than size (1) at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:305) at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:284) at org.apache.hadoop.hbase.regionserver.wal.LRUDictionary$BidirectionalLRUMap.get(LRUDictionary.java:124) at org.apache.hadoop.hbase.regionserver.wal.LRUDictionary$BidirectionalLRUMap.access$000(LRUDictionary.java:71) at org.apache.hadoop.hbase.regionserver.wal.LRUDictionary.getEntry(LRUDictionary.java:42) at org.apache.hadoop.hbase.regionserver.wal.WALCellCodec$CompressedKvDecoder.readIntoArray(WALCellCodec.java:210) at org.apache.hadoop.hbase.regionserver.wal.WALCellCodec$CompressedKvDecoder.parseCell(WALCellCodec.java:184) at org.apache.hadoop.hbase.codec.BaseDecoder.advance(BaseDecoder.java:46) at org.apache.hadoop.hbase.regionserver.wal.WALEdit.readFromCells(WALEdit.java:213) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:217) ... 4 more 2013-05-24 22:00:49,622 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(583): Nothing to replicate, sleeping 100 times 1 2013-05-24 22:00:49,723 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432840428 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:49,727 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 132235, fileLength: 132243, trailerPresent: true 2013-05-24 22:00:49,737 ERROR [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(236): Error while reading 691 WAL KVs; started reading at 53272 and read up to 65538 2013-05-24 22:00:49,738 WARN [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(324): 2 Got: java.io.IOException: Error while reading 691 WAL KVs; started reading at 53272 and read up to 65538 at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:237) at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:96) at org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.readNextAndSetPosition(ReplicationHLogReaderManager.java:89) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.readAllEntriesToReplicateOrNextFile(ReplicationSource.java:404) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:320) Caused by: java.lang.IndexOutOfBoundsException: index (30062) must be less than size (1) at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:305) at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:284) at org.apache.hadoop.hbase.regionserver.wal.LRUDictionary$BidirectionalLRUMap.get(LRUDictionary.java:124) at org.apache.hadoop.hbase.regionserver.wal.LRUDictionary$BidirectionalLRUMap.access$000(LRUDictionary.java:71) at org.apache.hadoop.hbase.regionserver.wal.LRUDictionary.getEntry(LRUDictionary.java:42) at org.apache.hadoop.hbase.regionserver.wal.WALCellCodec$CompressedKvDecoder.readIntoArray(WALCellCodec.java:210) at org.apache.hadoop.hbase.regionserver.wal.WALCellCodec$CompressedKvDecoder.parseCell(WALCellCodec.java:184) at org.apache.hadoop.hbase.codec.BaseDecoder.advance(BaseDecoder.java:46) at org.apache.hadoop.hbase.regionserver.wal.WALEdit.readFromCells(WALEdit.java:213) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:217) ... 4 more 2013-05-24 22:00:49,738 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(583): Nothing to replicate, sleeping 100 times 2 2013-05-24 22:00:49,791 INFO [kiyo.gq1.ygridcore.net,49701,1369432806997Replication Statistics #0] regionserver.Replication$ReplicationStatisticsThread(291): Normal source for cluster 2: Total replicated edits: 1629, currently replicating from: hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432832342 at position: N/A 2013-05-24 22:00:49,791 INFO [kiyo.gq1.ygridcore.net,42690,1369432806911Replication Statistics #0] regionserver.Replication$ReplicationStatisticsThread(291): Normal source for cluster 2: Total replicated edits: 2667, currently replicating from: hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432840428 at position: N/A 2013-05-24 22:00:49,940 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432840428 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:49,943 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 132235, fileLength: 132243, trailerPresent: true 2013-05-24 22:00:49,950 ERROR [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(236): Error while reading 691 WAL KVs; started reading at 53272 and read up to 65538 2013-05-24 22:00:49,950 WARN [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(324): 2 Got: java.io.IOException: Error while reading 691 WAL KVs; started reading at 53272 and read up to 65538 at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:237) at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:96) at org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.readNextAndSetPosition(ReplicationHLogReaderManager.java:89) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.readAllEntriesToReplicateOrNextFile(ReplicationSource.java:404) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:320) Caused by: java.lang.IndexOutOfBoundsException: index (30062) must be less than size (1) at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:305) at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:284) at org.apache.hadoop.hbase.regionserver.wal.LRUDictionary$BidirectionalLRUMap.get(LRUDictionary.java:124) at org.apache.hadoop.hbase.regionserver.wal.LRUDictionary$BidirectionalLRUMap.access$000(LRUDictionary.java:71) at org.apache.hadoop.hbase.regionserver.wal.LRUDictionary.getEntry(LRUDictionary.java:42) at org.apache.hadoop.hbase.regionserver.wal.WALCellCodec$CompressedKvDecoder.readIntoArray(WALCellCodec.java:210) at org.apache.hadoop.hbase.regionserver.wal.WALCellCodec$CompressedKvDecoder.parseCell(WALCellCodec.java:184) at org.apache.hadoop.hbase.codec.BaseDecoder.advance(BaseDecoder.java:46) at org.apache.hadoop.hbase.regionserver.wal.WALEdit.readFromCells(WALEdit.java:213) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:217) ... 4 more 2013-05-24 22:00:49,950 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(583): Nothing to replicate, sleeping 100 times 3 2013-05-24 22:00:50,253 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432840428 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:50,256 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 132235, fileLength: 132243, trailerPresent: true 2013-05-24 22:00:50,262 ERROR [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(236): Error while reading 691 WAL KVs; started reading at 53272 and read up to 65538 2013-05-24 22:00:50,263 WARN [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(324): 2 Got: java.io.IOException: Error while reading 691 WAL KVs; started reading at 53272 and read up to 65538 at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:237) at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:96) at org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.readNextAndSetPosition(ReplicationHLogReaderManager.java:89) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.readAllEntriesToReplicateOrNextFile(ReplicationSource.java:404) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:320) Caused by: java.lang.IndexOutOfBoundsException: index (30062) must be less than size (1) at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:305) at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:284) at org.apache.hadoop.hbase.regionserver.wal.LRUDictionary$BidirectionalLRUMap.get(LRUDictionary.java:124) at org.apache.hadoop.hbase.regionserver.wal.LRUDictionary$BidirectionalLRUMap.access$000(LRUDictionary.java:71) at org.apache.hadoop.hbase.regionserver.wal.LRUDictionary.getEntry(LRUDictionary.java:42) at org.apache.hadoop.hbase.regionserver.wal.WALCellCodec$CompressedKvDecoder.readIntoArray(WALCellCodec.java:210) at org.apache.hadoop.hbase.regionserver.wal.WALCellCodec$CompressedKvDecoder.parseCell(WALCellCodec.java:184) at org.apache.hadoop.hbase.codec.BaseDecoder.advance(BaseDecoder.java:46) at org.apache.hadoop.hbase.regionserver.wal.WALEdit.readFromCells(WALEdit.java:213) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:217) ... 4 more 2013-05-24 22:00:50,263 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(583): Nothing to replicate, sleeping 100 times 4 2013-05-24 22:00:50,471 DEBUG [Thread-584] client.ClientScanner(96): Creating scanner over test starting at key '' 2013-05-24 22:00:50,471 DEBUG [Thread-584] client.ClientScanner(209): Advancing internal scanner to startKey at '' 2013-05-24 22:00:50,546 DEBUG [Thread-584] client.ClientScanner(196): Finished scanning region {NAME => 'test,,1369432825000.45883ac87958f956040ff389c9f7eb86.', STARTKEY => '', ENDKEY => '', ENCODED => 45883ac87958f956040ff389c9f7eb86,} 2013-05-24 22:00:50,546 INFO [Thread-584] replication.TestReplicationQueueFailover(108): Only got 14304 rows instead of 17576 current i=0 2013-05-24 22:00:50,665 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432840428 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:50,669 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 132235, fileLength: 132243, trailerPresent: true 2013-05-24 22:00:50,678 ERROR [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(236): Error while reading 691 WAL KVs; started reading at 53272 and read up to 65538 2013-05-24 22:00:50,678 WARN [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(324): 2 Got: java.io.IOException: Error while reading 691 WAL KVs; started reading at 53272 and read up to 65538 at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:237) at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:96) at org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.readNextAndSetPosition(ReplicationHLogReaderManager.java:89) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.readAllEntriesToReplicateOrNextFile(ReplicationSource.java:404) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:320) Caused by: java.lang.IndexOutOfBoundsException: index (30062) must be less than size (1) at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:305) at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:284) at org.apache.hadoop.hbase.regionserver.wal.LRUDictionary$BidirectionalLRUMap.get(LRUDictionary.java:124) at org.apache.hadoop.hbase.regionserver.wal.LRUDictionary$BidirectionalLRUMap.access$000(LRUDictionary.java:71) at org.apache.hadoop.hbase.regionserver.wal.LRUDictionary.getEntry(LRUDictionary.java:42) at org.apache.hadoop.hbase.regionserver.wal.WALCellCodec$CompressedKvDecoder.readIntoArray(WALCellCodec.java:210) at org.apache.hadoop.hbase.regionserver.wal.WALCellCodec$CompressedKvDecoder.parseCell(WALCellCodec.java:184) at org.apache.hadoop.hbase.codec.BaseDecoder.advance(BaseDecoder.java:46) at org.apache.hadoop.hbase.regionserver.wal.WALEdit.readFromCells(WALEdit.java:213) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:217) ... 4 more 2013-05-24 22:00:50,678 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(583): Nothing to replicate, sleeping 100 times 5 2013-05-24 22:00:51,181 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432840428 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:51,185 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 132235, fileLength: 132243, trailerPresent: true 2013-05-24 22:00:51,194 ERROR [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(236): Error while reading 691 WAL KVs; started reading at 53272 and read up to 65538 2013-05-24 22:00:51,194 WARN [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(324): 2 Got: java.io.IOException: Error while reading 691 WAL KVs; started reading at 53272 and read up to 65538 at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:237) at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:96) at org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.readNextAndSetPosition(ReplicationHLogReaderManager.java:89) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.readAllEntriesToReplicateOrNextFile(ReplicationSource.java:404) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:320) Caused by: java.lang.IndexOutOfBoundsException: index (30062) must be less than size (1) at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:305) at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:284) at org.apache.hadoop.hbase.regionserver.wal.LRUDictionary$BidirectionalLRUMap.get(LRUDictionary.java:124) at org.apache.hadoop.hbase.regionserver.wal.LRUDictionary$BidirectionalLRUMap.access$000(LRUDictionary.java:71) at org.apache.hadoop.hbase.regionserver.wal.LRUDictionary.getEntry(LRUDictionary.java:42) at org.apache.hadoop.hbase.regionserver.wal.WALCellCodec$CompressedKvDecoder.readIntoArray(WALCellCodec.java:210) at org.apache.hadoop.hbase.regionserver.wal.WALCellCodec$CompressedKvDecoder.parseCell(WALCellCodec.java:184) at org.apache.hadoop.hbase.codec.BaseDecoder.advance(BaseDecoder.java:46) at org.apache.hadoop.hbase.regionserver.wal.WALEdit.readFromCells(WALEdit.java:213) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:217) ... 4 more 2013-05-24 22:00:51,194 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(583): Nothing to replicate, sleeping 100 times 6 2013-05-24 22:00:51,547 DEBUG [Thread-584] client.ClientScanner(96): Creating scanner over test starting at key '' 2013-05-24 22:00:51,547 DEBUG [Thread-584] client.ClientScanner(209): Advancing internal scanner to startKey at '' 2013-05-24 22:00:51,624 DEBUG [Thread-584] client.ClientScanner(196): Finished scanning region {NAME => 'test,,1369432825000.45883ac87958f956040ff389c9f7eb86.', STARTKEY => '', ENDKEY => '', ENCODED => 45883ac87958f956040ff389c9f7eb86,} 2013-05-24 22:00:51,624 INFO [Thread-584] replication.TestReplicationQueueFailover(108): Only got 14304 rows instead of 17576 current i=1 2013-05-24 22:00:51,796 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432840428 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:51,800 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 132235, fileLength: 132243, trailerPresent: true 2013-05-24 22:00:51,808 ERROR [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(236): Error while reading 691 WAL KVs; started reading at 53272 and read up to 65538 2013-05-24 22:00:51,808 WARN [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(324): 2 Got: java.io.IOException: Error while reading 691 WAL KVs; started reading at 53272 and read up to 65538 at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:237) at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:96) at org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.readNextAndSetPosition(ReplicationHLogReaderManager.java:89) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.readAllEntriesToReplicateOrNextFile(ReplicationSource.java:404) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:320) Caused by: java.lang.IndexOutOfBoundsException: index (30062) must be less than size (1) at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:305) at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:284) at org.apache.hadoop.hbase.regionserver.wal.LRUDictionary$BidirectionalLRUMap.get(LRUDictionary.java:124) at org.apache.hadoop.hbase.regionserver.wal.LRUDictionary$BidirectionalLRUMap.access$000(LRUDictionary.java:71) at org.apache.hadoop.hbase.regionserver.wal.LRUDictionary.getEntry(LRUDictionary.java:42) at org.apache.hadoop.hbase.regionserver.wal.WALCellCodec$CompressedKvDecoder.readIntoArray(WALCellCodec.java:210) at org.apache.hadoop.hbase.regionserver.wal.WALCellCodec$CompressedKvDecoder.parseCell(WALCellCodec.java:184) at org.apache.hadoop.hbase.codec.BaseDecoder.advance(BaseDecoder.java:46) at org.apache.hadoop.hbase.regionserver.wal.WALEdit.readFromCells(WALEdit.java:213) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:217) ... 4 more 2013-05-24 22:00:51,809 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(583): Nothing to replicate, sleeping 100 times 7 2013-05-24 22:00:52,511 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432840428 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:52,515 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 132235, fileLength: 132243, trailerPresent: true 2013-05-24 22:00:52,521 ERROR [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(236): Error while reading 691 WAL KVs; started reading at 53272 and read up to 65538 2013-05-24 22:00:52,521 WARN [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(324): 2 Got: java.io.IOException: Error while reading 691 WAL KVs; started reading at 53272 and read up to 65538 at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:237) at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:96) at org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.readNextAndSetPosition(ReplicationHLogReaderManager.java:89) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.readAllEntriesToReplicateOrNextFile(ReplicationSource.java:404) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:320) Caused by: java.lang.IndexOutOfBoundsException: index (30062) must be less than size (1) at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:305) at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:284) at org.apache.hadoop.hbase.regionserver.wal.LRUDictionary$BidirectionalLRUMap.get(LRUDictionary.java:124) at org.apache.hadoop.hbase.regionserver.wal.LRUDictionary$BidirectionalLRUMap.access$000(LRUDictionary.java:71) at org.apache.hadoop.hbase.regionserver.wal.LRUDictionary.getEntry(LRUDictionary.java:42) at org.apache.hadoop.hbase.regionserver.wal.WALCellCodec$CompressedKvDecoder.readIntoArray(WALCellCodec.java:210) at org.apache.hadoop.hbase.regionserver.wal.WALCellCodec$CompressedKvDecoder.parseCell(WALCellCodec.java:184) at org.apache.hadoop.hbase.codec.BaseDecoder.advance(BaseDecoder.java:46) at org.apache.hadoop.hbase.regionserver.wal.WALEdit.readFromCells(WALEdit.java:213) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:217) ... 4 more 2013-05-24 22:00:52,522 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(583): Nothing to replicate, sleeping 100 times 8 2013-05-24 22:00:52,624 DEBUG [Thread-584] client.ClientScanner(96): Creating scanner over test starting at key '' 2013-05-24 22:00:52,625 DEBUG [Thread-584] client.ClientScanner(209): Advancing internal scanner to startKey at '' 2013-05-24 22:00:52,698 DEBUG [Thread-584] client.ClientScanner(196): Finished scanning region {NAME => 'test,,1369432825000.45883ac87958f956040ff389c9f7eb86.', STARTKEY => '', ENDKEY => '', ENCODED => 45883ac87958f956040ff389c9f7eb86,} 2013-05-24 22:00:52,698 INFO [Thread-584] replication.TestReplicationQueueFailover(108): Only got 14304 rows instead of 17576 current i=2 2013-05-24 22:00:53,323 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432840428 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:53,328 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 132235, fileLength: 132243, trailerPresent: true 2013-05-24 22:00:53,377 INFO [kiyo.gq1.ygridcore.net,39146,1369432814418Replication Statistics #0] regionserver.Replication$ReplicationStatisticsThread(291): Sink: age in ms of last applied edit: 18282, total replicated edits: 2115 2013-05-24 22:00:53,382 ERROR [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(236): Error while reading 691 WAL KVs; started reading at 53272 and read up to 65538 2013-05-24 22:00:53,382 WARN [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(324): 2 Got: java.io.IOException: Error while reading 691 WAL KVs; started reading at 53272 and read up to 65538 at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:237) at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:96) at org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.readNextAndSetPosition(ReplicationHLogReaderManager.java:89) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.readAllEntriesToReplicateOrNextFile(ReplicationSource.java:404) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:320) Caused by: java.lang.IndexOutOfBoundsException: index (30062) must be less than size (1) at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:305) at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:284) at org.apache.hadoop.hbase.regionserver.wal.LRUDictionary$BidirectionalLRUMap.get(LRUDictionary.java:124) at org.apache.hadoop.hbase.regionserver.wal.LRUDictionary$BidirectionalLRUMap.access$000(LRUDictionary.java:71) at org.apache.hadoop.hbase.regionserver.wal.LRUDictionary.getEntry(LRUDictionary.java:42) at org.apache.hadoop.hbase.regionserver.wal.WALCellCodec$CompressedKvDecoder.readIntoArray(WALCellCodec.java:210) at org.apache.hadoop.hbase.regionserver.wal.WALCellCodec$CompressedKvDecoder.parseCell(WALCellCodec.java:184) at org.apache.hadoop.hbase.codec.BaseDecoder.advance(BaseDecoder.java:46) at org.apache.hadoop.hbase.regionserver.wal.WALEdit.readFromCells(WALEdit.java:213) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:217) ... 4 more 2013-05-24 22:00:53,382 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(583): Nothing to replicate, sleeping 100 times 9 2013-05-24 22:00:53,399 INFO [kiyo.gq1.ygridcore.net,51523,1369432814392Replication Statistics #0] regionserver.Replication$ReplicationStatisticsThread(291): Sink: age in ms of last applied edit: 11571, total replicated edits: 3593 2013-05-24 22:00:53,636 INFO [kiyo.gq1.ygridcore.net,53632,1369432814258.splitLogManagerTimeoutMonitor] master.SplitLogManager(604): Garbage collecting all recovering regions. 2013-05-24 22:00:53,699 DEBUG [Thread-584] client.ClientScanner(96): Creating scanner over test starting at key '' 2013-05-24 22:00:53,699 DEBUG [Thread-584] client.ClientScanner(209): Advancing internal scanner to startKey at '' 2013-05-24 22:00:53,777 DEBUG [Thread-584] client.ClientScanner(196): Finished scanning region {NAME => 'test,,1369432825000.45883ac87958f956040ff389c9f7eb86.', STARTKEY => '', ENDKEY => '', ENCODED => 45883ac87958f956040ff389c9f7eb86,} 2013-05-24 22:00:53,777 INFO [Thread-584] replication.TestReplicationQueueFailover(108): Only got 14304 rows instead of 17576 current i=3 2013-05-24 22:00:54,284 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432840428 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:54,288 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 132235, fileLength: 132243, trailerPresent: true 2013-05-24 22:00:54,296 ERROR [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(236): Error while reading 691 WAL KVs; started reading at 53272 and read up to 65538 2013-05-24 22:00:54,297 WARN [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(324): 2 Got: java.io.IOException: Error while reading 691 WAL KVs; started reading at 53272 and read up to 65538 at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:237) at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:96) at org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.readNextAndSetPosition(ReplicationHLogReaderManager.java:89) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.readAllEntriesToReplicateOrNextFile(ReplicationSource.java:404) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:320) Caused by: java.lang.IndexOutOfBoundsException: index (30062) must be less than size (1) at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:305) at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:284) at org.apache.hadoop.hbase.regionserver.wal.LRUDictionary$BidirectionalLRUMap.get(LRUDictionary.java:124) at org.apache.hadoop.hbase.regionserver.wal.LRUDictionary$BidirectionalLRUMap.access$000(LRUDictionary.java:71) at org.apache.hadoop.hbase.regionserver.wal.LRUDictionary.getEntry(LRUDictionary.java:42) at org.apache.hadoop.hbase.regionserver.wal.WALCellCodec$CompressedKvDecoder.readIntoArray(WALCellCodec.java:210) at org.apache.hadoop.hbase.regionserver.wal.WALCellCodec$CompressedKvDecoder.parseCell(WALCellCodec.java:184) at org.apache.hadoop.hbase.codec.BaseDecoder.advance(BaseDecoder.java:46) at org.apache.hadoop.hbase.regionserver.wal.WALEdit.readFromCells(WALEdit.java:213) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:217) ... 4 more 2013-05-24 22:00:54,297 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(583): Nothing to replicate, sleeping 100 times 10 2013-05-24 22:00:54,777 DEBUG [Thread-584] client.ClientScanner(96): Creating scanner over test starting at key '' 2013-05-24 22:00:54,778 DEBUG [Thread-584] client.ClientScanner(209): Advancing internal scanner to startKey at '' 2013-05-24 22:00:54,791 INFO [kiyo.gq1.ygridcore.net,42690,1369432806911Replication Statistics #0] regionserver.Replication$ReplicationStatisticsThread(291): Normal source for cluster 2: Total replicated edits: 2667, currently replicating from: hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432840428 at position: N/A 2013-05-24 22:00:54,791 INFO [kiyo.gq1.ygridcore.net,49701,1369432806997Replication Statistics #0] regionserver.Replication$ReplicationStatisticsThread(291): Normal source for cluster 2: Total replicated edits: 1629, currently replicating from: hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432832342 at position: N/A 2013-05-24 22:00:54,852 DEBUG [Thread-584] client.ClientScanner(196): Finished scanning region {NAME => 'test,,1369432825000.45883ac87958f956040ff389c9f7eb86.', STARTKEY => '', ENDKEY => '', ENCODED => 45883ac87958f956040ff389c9f7eb86,} 2013-05-24 22:00:54,852 INFO [Thread-584] replication.TestReplicationQueueFailover(108): Only got 14304 rows instead of 17576 current i=4 2013-05-24 22:00:55,299 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432840428 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:55,303 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 132235, fileLength: 132243, trailerPresent: true 2013-05-24 22:00:55,311 ERROR [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(236): Error while reading 691 WAL KVs; started reading at 53272 and read up to 65538 2013-05-24 22:00:55,312 WARN [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(324): 2 Got: java.io.IOException: Error while reading 691 WAL KVs; started reading at 53272 and read up to 65538 at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:237) at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:96) at org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.readNextAndSetPosition(ReplicationHLogReaderManager.java:89) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.readAllEntriesToReplicateOrNextFile(ReplicationSource.java:404) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:320) Caused by: java.lang.IndexOutOfBoundsException: index (30062) must be less than size (1) at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:305) at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:284) at org.apache.hadoop.hbase.regionserver.wal.LRUDictionary$BidirectionalLRUMap.get(LRUDictionary.java:124) at org.apache.hadoop.hbase.regionserver.wal.LRUDictionary$BidirectionalLRUMap.access$000(LRUDictionary.java:71) at org.apache.hadoop.hbase.regionserver.wal.LRUDictionary.getEntry(LRUDictionary.java:42) at org.apache.hadoop.hbase.regionserver.wal.WALCellCodec$CompressedKvDecoder.readIntoArray(WALCellCodec.java:210) at org.apache.hadoop.hbase.regionserver.wal.WALCellCodec$CompressedKvDecoder.parseCell(WALCellCodec.java:184) at org.apache.hadoop.hbase.codec.BaseDecoder.advance(BaseDecoder.java:46) at org.apache.hadoop.hbase.regionserver.wal.WALEdit.readFromCells(WALEdit.java:213) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:217) ... 4 more 2013-05-24 22:00:55,312 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(583): Nothing to replicate, sleeping 100 times 10 2013-05-24 22:00:55,852 DEBUG [Thread-584] client.ClientScanner(96): Creating scanner over test starting at key '' 2013-05-24 22:00:55,853 DEBUG [Thread-584] client.ClientScanner(209): Advancing internal scanner to startKey at '' 2013-05-24 22:00:55,926 DEBUG [Thread-584] client.ClientScanner(196): Finished scanning region {NAME => 'test,,1369432825000.45883ac87958f956040ff389c9f7eb86.', STARTKEY => '', ENDKEY => '', ENCODED => 45883ac87958f956040ff389c9f7eb86,} 2013-05-24 22:00:55,927 INFO [Thread-584] replication.TestReplicationQueueFailover(108): Only got 14304 rows instead of 17576 current i=5 2013-05-24 22:00:56,314 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432840428 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:56,318 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 132235, fileLength: 132243, trailerPresent: true 2013-05-24 22:00:56,327 ERROR [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(236): Error while reading 691 WAL KVs; started reading at 53272 and read up to 65538 2013-05-24 22:00:56,327 WARN [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(324): 2 Got: java.io.IOException: Error while reading 691 WAL KVs; started reading at 53272 and read up to 65538 at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:237) at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:96) at org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.readNextAndSetPosition(ReplicationHLogReaderManager.java:89) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.readAllEntriesToReplicateOrNextFile(ReplicationSource.java:404) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:320) Caused by: java.lang.IndexOutOfBoundsException: index (30062) must be less than size (1) at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:305) at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:284) at org.apache.hadoop.hbase.regionserver.wal.LRUDictionary$BidirectionalLRUMap.get(LRUDictionary.java:124) at org.apache.hadoop.hbase.regionserver.wal.LRUDictionary$BidirectionalLRUMap.access$000(LRUDictionary.java:71) at org.apache.hadoop.hbase.regionserver.wal.LRUDictionary.getEntry(LRUDictionary.java:42) at org.apache.hadoop.hbase.regionserver.wal.WALCellCodec$CompressedKvDecoder.readIntoArray(WALCellCodec.java:210) at org.apache.hadoop.hbase.regionserver.wal.WALCellCodec$CompressedKvDecoder.parseCell(WALCellCodec.java:184) at org.apache.hadoop.hbase.codec.BaseDecoder.advance(BaseDecoder.java:46) at org.apache.hadoop.hbase.regionserver.wal.WALEdit.readFromCells(WALEdit.java:213) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:217) ... 4 more 2013-05-24 22:00:56,328 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(583): Nothing to replicate, sleeping 100 times 10 2013-05-24 22:00:56,927 DEBUG [Thread-584] client.ClientScanner(96): Creating scanner over test starting at key '' 2013-05-24 22:00:56,927 DEBUG [Thread-584] client.ClientScanner(209): Advancing internal scanner to startKey at '' 2013-05-24 22:00:56,999 DEBUG [Thread-584] client.ClientScanner(196): Finished scanning region {NAME => 'test,,1369432825000.45883ac87958f956040ff389c9f7eb86.', STARTKEY => '', ENDKEY => '', ENCODED => 45883ac87958f956040ff389c9f7eb86,} 2013-05-24 22:00:56,999 INFO [Thread-584] replication.TestReplicationQueueFailover(108): Only got 14304 rows instead of 17576 current i=6 2013-05-24 22:00:57,330 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432840428 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:57,333 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 132235, fileLength: 132243, trailerPresent: true 2013-05-24 22:00:57,339 ERROR [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(236): Error while reading 691 WAL KVs; started reading at 53272 and read up to 65538 2013-05-24 22:00:57,339 WARN [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(324): 2 Got: java.io.IOException: Error while reading 691 WAL KVs; started reading at 53272 and read up to 65538 at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:237) at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:96) at org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.readNextAndSetPosition(ReplicationHLogReaderManager.java:89) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.readAllEntriesToReplicateOrNextFile(ReplicationSource.java:404) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:320) Caused by: java.lang.IndexOutOfBoundsException: index (30062) must be less than size (1) at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:305) at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:284) at org.apache.hadoop.hbase.regionserver.wal.LRUDictionary$BidirectionalLRUMap.get(LRUDictionary.java:124) at org.apache.hadoop.hbase.regionserver.wal.LRUDictionary$BidirectionalLRUMap.access$000(LRUDictionary.java:71) at org.apache.hadoop.hbase.regionserver.wal.LRUDictionary.getEntry(LRUDictionary.java:42) at org.apache.hadoop.hbase.regionserver.wal.WALCellCodec$CompressedKvDecoder.readIntoArray(WALCellCodec.java:210) at org.apache.hadoop.hbase.regionserver.wal.WALCellCodec$CompressedKvDecoder.parseCell(WALCellCodec.java:184) at org.apache.hadoop.hbase.codec.BaseDecoder.advance(BaseDecoder.java:46) at org.apache.hadoop.hbase.regionserver.wal.WALEdit.readFromCells(WALEdit.java:213) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:217) ... 4 more 2013-05-24 22:00:57,339 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(583): Nothing to replicate, sleeping 100 times 10 2013-05-24 22:00:57,999 DEBUG [Thread-584] client.ClientScanner(96): Creating scanner over test starting at key '' 2013-05-24 22:00:58,000 DEBUG [Thread-584] client.ClientScanner(209): Advancing internal scanner to startKey at '' 2013-05-24 22:00:58,076 DEBUG [Thread-584] client.ClientScanner(196): Finished scanning region {NAME => 'test,,1369432825000.45883ac87958f956040ff389c9f7eb86.', STARTKEY => '', ENDKEY => '', ENCODED => 45883ac87958f956040ff389c9f7eb86,} 2013-05-24 22:00:58,076 INFO [Thread-584] replication.TestReplicationQueueFailover(108): Only got 14304 rows instead of 17576 current i=7 2013-05-24 22:00:58,341 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432840428 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:58,343 INFO [kiyo.gq1.ygridcore.net,39146,1369432814418Replication Statistics #0] regionserver.Replication$ReplicationStatisticsThread(291): Sink: age in ms of last applied edit: 23248, total replicated edits: 2115 2013-05-24 22:00:58,345 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 132235, fileLength: 132243, trailerPresent: true 2013-05-24 22:00:58,353 ERROR [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(236): Error while reading 691 WAL KVs; started reading at 53272 and read up to 65538 2013-05-24 22:00:58,353 WARN [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(324): 2 Got: java.io.IOException: Error while reading 691 WAL KVs; started reading at 53272 and read up to 65538 at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:237) at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:96) at org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.readNextAndSetPosition(ReplicationHLogReaderManager.java:89) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.readAllEntriesToReplicateOrNextFile(ReplicationSource.java:404) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:320) Caused by: java.lang.IndexOutOfBoundsException: index (30062) must be less than size (1) at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:305) at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:284) at org.apache.hadoop.hbase.regionserver.wal.LRUDictionary$BidirectionalLRUMap.get(LRUDictionary.java:124) at org.apache.hadoop.hbase.regionserver.wal.LRUDictionary$BidirectionalLRUMap.access$000(LRUDictionary.java:71) at org.apache.hadoop.hbase.regionserver.wal.LRUDictionary.getEntry(LRUDictionary.java:42) at org.apache.hadoop.hbase.regionserver.wal.WALCellCodec$CompressedKvDecoder.readIntoArray(WALCellCodec.java:210) at org.apache.hadoop.hbase.regionserver.wal.WALCellCodec$CompressedKvDecoder.parseCell(WALCellCodec.java:184) at org.apache.hadoop.hbase.codec.BaseDecoder.advance(BaseDecoder.java:46) at org.apache.hadoop.hbase.regionserver.wal.WALEdit.readFromCells(WALEdit.java:213) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:217) ... 4 more 2013-05-24 22:00:58,354 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(583): Nothing to replicate, sleeping 100 times 10 2013-05-24 22:00:58,399 INFO [kiyo.gq1.ygridcore.net,51523,1369432814392Replication Statistics #0] regionserver.Replication$ReplicationStatisticsThread(291): Sink: age in ms of last applied edit: 16571, total replicated edits: 3593 2013-05-24 22:00:59,077 DEBUG [Thread-584] client.ClientScanner(96): Creating scanner over test starting at key '' 2013-05-24 22:00:59,077 DEBUG [Thread-584] client.ClientScanner(209): Advancing internal scanner to startKey at '' 2013-05-24 22:00:59,159 DEBUG [Thread-584] client.ClientScanner(196): Finished scanning region {NAME => 'test,,1369432825000.45883ac87958f956040ff389c9f7eb86.', STARTKEY => '', ENDKEY => '', ENCODED => 45883ac87958f956040ff389c9f7eb86,} 2013-05-24 22:00:59,160 INFO [Thread-584] replication.TestReplicationQueueFailover(108): Only got 14304 rows instead of 17576 current i=8 2013-05-24 22:00:59,356 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432840428 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:00:59,360 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 132235, fileLength: 132243, trailerPresent: true 2013-05-24 22:00:59,368 ERROR [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(236): Error while reading 691 WAL KVs; started reading at 53272 and read up to 65538 2013-05-24 22:00:59,368 WARN [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(324): 2 Got: java.io.IOException: Error while reading 691 WAL KVs; started reading at 53272 and read up to 65538 at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:237) at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:96) at org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.readNextAndSetPosition(ReplicationHLogReaderManager.java:89) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.readAllEntriesToReplicateOrNextFile(ReplicationSource.java:404) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:320) Caused by: java.lang.IndexOutOfBoundsException: index (30062) must be less than size (1) at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:305) at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:284) at org.apache.hadoop.hbase.regionserver.wal.LRUDictionary$BidirectionalLRUMap.get(LRUDictionary.java:124) at org.apache.hadoop.hbase.regionserver.wal.LRUDictionary$BidirectionalLRUMap.access$000(LRUDictionary.java:71) at org.apache.hadoop.hbase.regionserver.wal.LRUDictionary.getEntry(LRUDictionary.java:42) at org.apache.hadoop.hbase.regionserver.wal.WALCellCodec$CompressedKvDecoder.readIntoArray(WALCellCodec.java:210) at org.apache.hadoop.hbase.regionserver.wal.WALCellCodec$CompressedKvDecoder.parseCell(WALCellCodec.java:184) at org.apache.hadoop.hbase.codec.BaseDecoder.advance(BaseDecoder.java:46) at org.apache.hadoop.hbase.regionserver.wal.WALEdit.readFromCells(WALEdit.java:213) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:217) ... 4 more 2013-05-24 22:00:59,368 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(583): Nothing to replicate, sleeping 100 times 10 2013-05-24 22:00:59,791 INFO [kiyo.gq1.ygridcore.net,42690,1369432806911Replication Statistics #0] regionserver.Replication$ReplicationStatisticsThread(291): Normal source for cluster 2: Total replicated edits: 2667, currently replicating from: hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432840428 at position: N/A 2013-05-24 22:00:59,791 INFO [kiyo.gq1.ygridcore.net,49701,1369432806997Replication Statistics #0] regionserver.Replication$ReplicationStatisticsThread(291): Normal source for cluster 2: Total replicated edits: 1629, currently replicating from: hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432832342 at position: N/A 2013-05-24 22:01:00,160 DEBUG [Thread-584] client.ClientScanner(96): Creating scanner over test starting at key '' 2013-05-24 22:01:00,160 DEBUG [Thread-584] client.ClientScanner(209): Advancing internal scanner to startKey at '' 2013-05-24 22:01:00,239 DEBUG [Thread-584] client.ClientScanner(196): Finished scanning region {NAME => 'test,,1369432825000.45883ac87958f956040ff389c9f7eb86.', STARTKEY => '', ENDKEY => '', ENCODED => 45883ac87958f956040ff389c9f7eb86,} 2013-05-24 22:01:00,240 INFO [Thread-584] replication.TestReplicationQueueFailover(108): Only got 14304 rows instead of 17576 current i=9 2013-05-24 22:01:00,371 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432840428 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:01:00,375 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 132235, fileLength: 132243, trailerPresent: true 2013-05-24 22:01:00,381 ERROR [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(236): Error while reading 691 WAL KVs; started reading at 53272 and read up to 65538 2013-05-24 22:01:00,381 WARN [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(324): 2 Got: java.io.IOException: Error while reading 691 WAL KVs; started reading at 53272 and read up to 65538 at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:237) at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:96) at org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.readNextAndSetPosition(ReplicationHLogReaderManager.java:89) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.readAllEntriesToReplicateOrNextFile(ReplicationSource.java:404) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:320) Caused by: java.lang.IndexOutOfBoundsException: index (30062) must be less than size (1) at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:305) at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:284) at org.apache.hadoop.hbase.regionserver.wal.LRUDictionary$BidirectionalLRUMap.get(LRUDictionary.java:124) at org.apache.hadoop.hbase.regionserver.wal.LRUDictionary$BidirectionalLRUMap.access$000(LRUDictionary.java:71) at org.apache.hadoop.hbase.regionserver.wal.LRUDictionary.getEntry(LRUDictionary.java:42) at org.apache.hadoop.hbase.regionserver.wal.WALCellCodec$CompressedKvDecoder.readIntoArray(WALCellCodec.java:210) at org.apache.hadoop.hbase.regionserver.wal.WALCellCodec$CompressedKvDecoder.parseCell(WALCellCodec.java:184) at org.apache.hadoop.hbase.codec.BaseDecoder.advance(BaseDecoder.java:46) at org.apache.hadoop.hbase.regionserver.wal.WALEdit.readFromCells(WALEdit.java:213) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:217) ... 4 more 2013-05-24 22:01:00,382 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(583): Nothing to replicate, sleeping 100 times 10 2013-05-24 22:01:01,281 INFO [pool-1-thread-1] hbase.ResourceChecker(171): after: replication.TestReplicationQueueFailoverCompressed#queueFailover Thread=414 (was 513), OpenFileDescriptor=568 (was 701), MaxFileDescriptor=32768 (was 32768), SystemLoadAverage=819 (was 835), ProcessCount=483 (was 483), AvailableMemoryMB=6378 (was 6810), ConnectionCount=6 (was 8) 2013-05-24 22:01:01,282 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(911): Shutting down minicluster 2013-05-24 22:01:01,283 DEBUG [pool-1-thread-1] util.JVMClusterUtil(237): Shutting down HBase Cluster 2013-05-24 22:01:01,284 INFO [pool-1-thread-1] master.HMaster(2216): Cluster shutdown requested 2013-05-24 22:01:01,284 INFO [kiyo.gq1.ygridcore.net,53632,1369432814258-ClusterStatusChore] hbase.Chore(93): kiyo.gq1.ygridcore.net,53632,1369432814258-ClusterStatusChore exiting 2013-05-24 22:01:01,284 INFO [kiyo.gq1.ygridcore.net,53632,1369432814258-BalancerChore] hbase.Chore(93): kiyo.gq1.ygridcore.net,53632,1369432814258-BalancerChore exiting 2013-05-24 22:01:01,284 INFO [kiyo.gq1.ygridcore.net,53632,1369432814258-CatalogJanitor] hbase.Chore(93): kiyo.gq1.ygridcore.net,53632,1369432814258-CatalogJanitor exiting 2013-05-24 22:01:01,284 INFO [Master:0;kiyo.gq1.ygridcore.net,53632,1369432814258] master.ServerManager(448): Waiting on regionserver(s) to go down kiyo.gq1.ygridcore.net,51523,1369432814392 2013-05-24 22:01:01,352 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:51523-0x13ed88e2cdf0010 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/2/running 2013-05-24 22:01:01,353 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:53632-0x13ed88e2cdf000f Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/2/running 2013-05-24 22:01:01,353 INFO [pool-1-thread-1] regionserver.HRegionServer(1639): STOPPED: Shutdown requested 2013-05-24 22:01:01,353 INFO [pool-1-thread-1] regionserver.HRegionServer(1639): STOPPED: Shutdown requested 2013-05-24 22:01:01,354 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392-EventThread] zookeeper.ZKUtil(433): regionserver:51523-0x13ed88e2cdf0010 Set watcher on znode that does not yet exist, /2/running 2013-05-24 22:01:01,355 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392] regionserver.SplitLogWorker(585): Sending interrupt to stop the worker thread 2013-05-24 22:01:01,355 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] regionserver.SplitLogWorker(267): SplitLogWorker interrupted while waiting for task, exiting: java.lang.InterruptedException 2013-05-24 22:01:01,356 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392] snapshot.RegionServerSnapshotManager(151): Stopping RegionServerSnapshotManager gracefully. 2013-05-24 22:01:01,356 INFO [RS_OPEN_META-kiyo.gq1.ygridcore.net,51523,1369432814392-0MetaLogRoller] regionserver.LogRoller(119): LogRoller exiting. 2013-05-24 22:01:01,356 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,51523,1369432814392] regionserver.SplitLogWorker(199): SplitLogWorker kiyo.gq1.ygridcore.net,51523,1369432814392 exiting 2013-05-24 22:01:01,356 INFO [Thread-344] regionserver.MemStoreFlusher$FlushHandler(267): Thread-344 exiting 2013-05-24 22:01:01,356 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392.logRoller] regionserver.LogRoller(119): LogRoller exiting. 2013-05-24 22:01:01,356 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392.compactionChecker] hbase.Chore(93): RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392.compactionChecker exiting 2013-05-24 22:01:01,356 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(433): master:53632-0x13ed88e2cdf000f Set watcher on znode that does not yet exist, /2/running 2013-05-24 22:01:01,357 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392] regionserver.HRegionServer(886): stopping server kiyo.gq1.ygridcore.net,51523,1369432814392 2013-05-24 22:01:01,357 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,51523,1369432814392-0] handler.CloseRegionHandler(125): Processing close of test,,1369432825000.45883ac87958f956040ff389c9f7eb86. 2013-05-24 22:01:01,357 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392] catalog.CatalogTracker(208): Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@6cbbe73b 2013-05-24 22:01:01,357 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,51523,1369432814392-0] regionserver.HRegion(927): Closing test,,1369432825000.45883ac87958f956040ff389c9f7eb86.: disabling compactions & flushes 2013-05-24 22:01:01,357 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392] client.HConnectionManager$HConnectionImplementation(1640): Closing zookeeper sessionid=0x13ed88e2cdf0014 2013-05-24 22:01:01,357 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,51523,1369432814392-0] regionserver.HRegion(949): Updates disabled for region test,,1369432825000.45883ac87958f956040ff389c9f7eb86. 2013-05-24 22:01:01,358 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,51523,1369432814392-0] regionserver.HRegion(1452): Started memstore flush for test,,1369432825000.45883ac87958f956040ff389c9f7eb86., current region memstore size 2.7 M 2013-05-24 22:01:01,384 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432840428 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:01:01,385 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392.periodicFlusher] hbase.Chore(93): RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392.periodicFlusher exiting 2013-05-24 22:01:01,388 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 132235, fileLength: 132243, trailerPresent: true 2013-05-24 22:01:01,396 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392.leaseChecker] regionserver.Leases(124): RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392.leaseChecker closing leases 2013-05-24 22:01:01,397 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392.leaseChecker] regionserver.Leases(131): RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392.leaseChecker closed leases 2013-05-24 22:01:01,419 ERROR [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(236): Error while reading 691 WAL KVs; started reading at 53272 and read up to 65538 2013-05-24 22:01:01,420 WARN [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(324): 2 Got: java.io.IOException: Error while reading 691 WAL KVs; started reading at 53272 and read up to 65538 at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:237) at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:96) at org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.readNextAndSetPosition(ReplicationHLogReaderManager.java:89) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.readAllEntriesToReplicateOrNextFile(ReplicationSource.java:404) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:320) Caused by: java.lang.IndexOutOfBoundsException: index (30062) must be less than size (1) at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:305) at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:284) at org.apache.hadoop.hbase.regionserver.wal.LRUDictionary$BidirectionalLRUMap.get(LRUDictionary.java:124) at org.apache.hadoop.hbase.regionserver.wal.LRUDictionary$BidirectionalLRUMap.access$000(LRUDictionary.java:71) at org.apache.hadoop.hbase.regionserver.wal.LRUDictionary.getEntry(LRUDictionary.java:42) at org.apache.hadoop.hbase.regionserver.wal.WALCellCodec$CompressedKvDecoder.readIntoArray(WALCellCodec.java:210) at org.apache.hadoop.hbase.regionserver.wal.WALCellCodec$CompressedKvDecoder.parseCell(WALCellCodec.java:184) at org.apache.hadoop.hbase.codec.BaseDecoder.advance(BaseDecoder.java:46) at org.apache.hadoop.hbase.regionserver.wal.WALEdit.readFromCells(WALEdit.java:213) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:217) ... 4 more 2013-05-24 22:01:01,422 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(583): Nothing to replicate, sleeping 100 times 10 2013-05-24 22:01:01,494 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392] snapshot.RegionServerSnapshotManager(151): Stopping RegionServerSnapshotManager gracefully. 2013-05-24 22:01:01,494 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392] regionserver.CompactSplitThread(318): Waiting for Split Thread to finish... 2013-05-24 22:01:01,494 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392] regionserver.CompactSplitThread(318): Waiting for Merge Thread to finish... 2013-05-24 22:01:01,494 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392] regionserver.CompactSplitThread(318): Waiting for Large Compaction Thread to finish... 2013-05-24 22:01:01,494 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392] regionserver.CompactSplitThread(318): Waiting for Small Compaction Thread to finish... 2013-05-24 22:01:01,495 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392] regionserver.HRegionServer(1048): Waiting on 2 regions to close 2013-05-24 22:01:01,495 DEBUG [RS_CLOSE_META-kiyo.gq1.ygridcore.net,51523,1369432814392-0] handler.CloseRegionHandler(125): Processing close of .META.,,1.1028785192 2013-05-24 22:01:01,495 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392] regionserver.HRegionServer(1052): {1028785192=.META.,,1.1028785192, 45883ac87958f956040ff389c9f7eb86=test,,1369432825000.45883ac87958f956040ff389c9f7eb86.} 2013-05-24 22:01:01,495 DEBUG [RS_CLOSE_META-kiyo.gq1.ygridcore.net,51523,1369432814392-0] regionserver.HRegion(927): Closing .META.,,1.1028785192: disabling compactions & flushes 2013-05-24 22:01:01,496 DEBUG [RS_CLOSE_META-kiyo.gq1.ygridcore.net,51523,1369432814392-0] regionserver.HRegion(949): Updates disabled for region .META.,,1.1028785192 2013-05-24 22:01:01,496 DEBUG [RS_CLOSE_META-kiyo.gq1.ygridcore.net,51523,1369432814392-0] regionserver.HRegion(1452): Started memstore flush for .META.,,1.1028785192, current region memstore size 1.7 K 2013-05-24 22:01:01,633 INFO [IPC Server handler 7 on 54104] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52910 is added to blk_895790808292773443_1022{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:33054|RBW], ReplicaUnderConstruction[127.0.0.1:52910|RBW]]} size 0 2013-05-24 22:01:01,634 INFO [IPC Server handler 5 on 54104] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33054 is added to blk_895790808292773443_1022 size 1615 2013-05-24 22:01:01,636 INFO [kiyo.gq1.ygridcore.net,53632,1369432814258.splitLogManagerTimeoutMonitor] hbase.Chore(93): kiyo.gq1.ygridcore.net,53632,1369432814258.splitLogManagerTimeoutMonitor exiting 2013-05-24 22:01:01,727 INFO [RS_CLOSE_META-kiyo.gq1.ygridcore.net,51523,1369432814392-0] regionserver.DefaultStoreFlusher(88): Flushed, sequenceid=5, memsize=1.7 K, hasBloomFilter=false, into tmp file hdfs://localhost:54104/user/hortonzy/hbase/.META./1028785192/.tmp/9521c8ee9f09423b86129db40efdd543 2013-05-24 22:01:01,759 DEBUG [RS_CLOSE_META-kiyo.gq1.ygridcore.net,51523,1369432814392-0] regionserver.HRegionFileSystem(338): Committing store file hdfs://localhost:54104/user/hortonzy/hbase/.META./1028785192/.tmp/9521c8ee9f09423b86129db40efdd543 as hdfs://localhost:54104/user/hortonzy/hbase/.META./1028785192/info/9521c8ee9f09423b86129db40efdd543 2013-05-24 22:01:01,772 INFO [IPC Server handler 3 on 54104] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33054 is added to blk_-6590034066981360352_1023{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:52910|RBW], ReplicaUnderConstruction[127.0.0.1:33054|RBW]]} size 0 2013-05-24 22:01:01,772 INFO [IPC Server handler 1 on 54104] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52910 is added to blk_-6590034066981360352_1023{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:52910|RBW], ReplicaUnderConstruction[127.0.0.1:33054|RBW]]} size 0 2013-05-24 22:01:01,800 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,51523,1369432814392-0] regionserver.DefaultStoreFlusher(88): Flushed, sequenceid=15, memsize=2.3 M, hasBloomFilter=true, into tmp file hdfs://localhost:54104/user/hortonzy/hbase/test/45883ac87958f956040ff389c9f7eb86/.tmp/a96f9dc03fe84310a5d3f84f6d5c243e 2013-05-24 22:01:01,808 INFO [RS_CLOSE_META-kiyo.gq1.ygridcore.net,51523,1369432814392-0] regionserver.HStore(761): Added hdfs://localhost:54104/user/hortonzy/hbase/.META./1028785192/info/9521c8ee9f09423b86129db40efdd543, entries=7, sequenceid=5, filesize=1.6 K 2013-05-24 22:01:01,808 INFO [RS_CLOSE_META-kiyo.gq1.ygridcore.net,51523,1369432814392-0] regionserver.HRegion(1596): Finished memstore flush of ~1.7 K/1712, currentsize=0/0 for region .META.,,1.1028785192 in 312ms, sequenceid=5, compaction requested=false 2013-05-24 22:01:01,814 INFO [StoreCloserThread-.META.,,1.1028785192-1] regionserver.HStore(663): Closed info 2013-05-24 22:01:01,815 INFO [RS_CLOSE_META-kiyo.gq1.ygridcore.net,51523,1369432814392-0] regionserver.HRegion(1006): Closed .META.,,1.1028785192 2013-05-24 22:01:01,815 DEBUG [RS_CLOSE_META-kiyo.gq1.ygridcore.net,51523,1369432814392-0] handler.CloseRegionHandler(177): Closed region .META.,,1.1028785192 2013-05-24 22:01:01,816 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,51523,1369432814392-0] regionserver.HRegionFileSystem(338): Committing store file hdfs://localhost:54104/user/hortonzy/hbase/test/45883ac87958f956040ff389c9f7eb86/.tmp/a96f9dc03fe84310a5d3f84f6d5c243e as hdfs://localhost:54104/user/hortonzy/hbase/test/45883ac87958f956040ff389c9f7eb86/f/a96f9dc03fe84310a5d3f84f6d5c243e 2013-05-24 22:01:01,840 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,51523,1369432814392-0] regionserver.HStore(761): Added hdfs://localhost:54104/user/hortonzy/hbase/test/45883ac87958f956040ff389c9f7eb86/f/a96f9dc03fe84310a5d3f84f6d5c243e, entries=14304, sequenceid=15, filesize=424.6 K 2013-05-24 22:01:01,841 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,51523,1369432814392-0] regionserver.HRegion(1596): Finished memstore flush of ~2.7 M/2868264, currentsize=0/0 for region test,,1369432825000.45883ac87958f956040ff389c9f7eb86. in 483ms, sequenceid=15, compaction requested=false 2013-05-24 22:01:01,845 INFO [StoreCloserThread-test,,1369432825000.45883ac87958f956040ff389c9f7eb86.-1] regionserver.HStore(663): Closed f 2013-05-24 22:01:01,845 INFO [StoreCloserThread-test,,1369432825000.45883ac87958f956040ff389c9f7eb86.-1] regionserver.HStore(663): Closed norep 2013-05-24 22:01:01,845 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,51523,1369432814392-0] regionserver.HRegion(1006): Closed test,,1369432825000.45883ac87958f956040ff389c9f7eb86. 2013-05-24 22:01:01,845 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,51523,1369432814392-0] handler.CloseRegionHandler(177): Closed region test,,1369432825000.45883ac87958f956040ff389c9f7eb86. 2013-05-24 22:01:01,896 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392] regionserver.HRegionServer(912): stopping server kiyo.gq1.ygridcore.net,51523,1369432814392; all regions closed. 2013-05-24 22:01:01,896 INFO [RS_OPEN_META-kiyo.gq1.ygridcore.net,51523,1369432814392-0.logSyncer] wal.FSHLog$LogSyncer(967): RS_OPEN_META-kiyo.gq1.ygridcore.net,51523,1369432814392-0.logSyncer exiting 2013-05-24 22:01:01,896 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392] wal.FSHLog(809): closing hlog writer in hdfs://localhost:54104/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,51523,1369432814392 2013-05-24 22:01:01,898 INFO [IPC Server handler 8 on 54104] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52910 is added to blk_-673267344410927344_1008{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:33054|RBW], ReplicaUnderConstruction[127.0.0.1:52910|RBW]]} size 0 2013-05-24 22:01:01,899 INFO [IPC Server handler 2 on 54104] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33054 is added to blk_-673267344410927344_1008 size 429 2013-05-24 22:01:01,925 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392.logSyncer] wal.FSHLog$LogSyncer(967): RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392.logSyncer exiting 2013-05-24 22:01:01,926 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392] wal.FSHLog(809): closing hlog writer in hdfs://localhost:54104/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,51523,1369432814392 2013-05-24 22:01:01,932 INFO [IPC Server handler 5 on 54104] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33054 is added to blk_-4299950353232388429_1024{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:52910|RBW], ReplicaUnderConstruction[127.0.0.1:33054|RBW]]} size 0 2013-05-24 22:01:01,933 INFO [IPC Server handler 6 on 54104] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52910 is added to blk_-4299950353232388429_1024 size 15 2013-05-24 22:01:02,351 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392] wal.FSHLog(768): Moved 9 log files to /user/hortonzy/hbase/.oldlogs 2013-05-24 22:01:02,385 INFO [Master:0;kiyo.gq1.ygridcore.net,53632,1369432814258] master.ServerManager(448): Waiting on regionserver(s) to go down kiyo.gq1.ygridcore.net,51523,1369432814392 2013-05-24 22:01:02,424 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432840428 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net 2013-05-24 22:01:02,429 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 132235, fileLength: 132243, trailerPresent: true 2013-05-24 22:01:02,438 ERROR [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(236): Error while reading 691 WAL KVs; started reading at 53272 and read up to 65538 2013-05-24 22:01:02,438 WARN [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(324): 2 Got: java.io.IOException: Error while reading 691 WAL KVs; started reading at 53272 and read up to 65538 at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:237) at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:96) at org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.readNextAndSetPosition(ReplicationHLogReaderManager.java:89) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.readAllEntriesToReplicateOrNextFile(ReplicationSource.java:404) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:320) Caused by: java.lang.IndexOutOfBoundsException: index (30062) must be less than size (1) at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:305) at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:284) at org.apache.hadoop.hbase.regionserver.wal.LRUDictionary$BidirectionalLRUMap.get(LRUDictionary.java:124) at org.apache.hadoop.hbase.regionserver.wal.LRUDictionary$BidirectionalLRUMap.access$000(LRUDictionary.java:71) at org.apache.hadoop.hbase.regionserver.wal.LRUDictionary.getEntry(LRUDictionary.java:42) at org.apache.hadoop.hbase.regionserver.wal.WALCellCodec$CompressedKvDecoder.readIntoArray(WALCellCodec.java:210) at org.apache.hadoop.hbase.regionserver.wal.WALCellCodec$CompressedKvDecoder.parseCell(WALCellCodec.java:184) at org.apache.hadoop.hbase.codec.BaseDecoder.advance(BaseDecoder.java:46) at org.apache.hadoop.hbase.regionserver.wal.WALEdit.readFromCells(WALEdit.java:213) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:217) ... 4 more 2013-05-24 22:01:02,439 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(583): Nothing to replicate, sleeping 100 times 10 2013-05-24 22:01:02,483 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392] regionserver.Leases(124): RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392 closing leases 2013-05-24 22:01:02,483 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392] regionserver.Leases(131): RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392 closed leases 2013-05-24 22:01:02,505 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392] client.HConnectionManager$HConnectionImplementation(1640): Closing zookeeper sessionid=0x13ed88e2cdf0017 2013-05-24 22:01:02,551 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:51523-0x13ed88e2cdf0010 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/2/rs/kiyo.gq1.ygridcore.net,51523,1369432814392 2013-05-24 22:01:02,551 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:53632-0x13ed88e2cdf000f Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/2/rs/kiyo.gq1.ygridcore.net,51523,1369432814392 2013-05-24 22:01:02,551 INFO [pool-1-thread-1-EventThread] zookeeper.RegionServerTracker(94): RegionServer ephemeral node deleted, processing expiration [kiyo.gq1.ygridcore.net,51523,1369432814392] 2013-05-24 22:01:02,551 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:51523-0x13ed88e2cdf0010 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/2/rs 2013-05-24 22:01:02,552 INFO [pool-1-thread-1-EventThread] master.ServerManager(495): Cluster shutdown set; kiyo.gq1.ygridcore.net,51523,1369432814392 expired; onlineServers=0 2013-05-24 22:01:02,552 INFO [pool-1-thread-1-EventThread] master.HMaster(2216): Cluster shutdown set; onlineServer=0 2013-05-24 22:01:02,552 DEBUG [pool-1-thread-1-EventThread] catalog.CatalogTracker(208): Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@24aaa392 2013-05-24 22:01:02,552 DEBUG [Master:0;kiyo.gq1.ygridcore.net,53632,1369432814258] master.HMaster(1111): Stopping service threads 2013-05-24 22:01:02,552 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:53632-0x13ed88e2cdf000f Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/2/rs 2013-05-24 22:01:02,554 INFO [Master:0;kiyo.gq1.ygridcore.net,53632,1369432814258.oldLogCleaner] hbase.Chore(93): Master:0;kiyo.gq1.ygridcore.net,53632,1369432814258.oldLogCleaner exiting 2013-05-24 22:01:02,554 INFO [Master:0;kiyo.gq1.ygridcore.net,53632,1369432814258.archivedHFileCleaner] hbase.Chore(93): Master:0;kiyo.gq1.ygridcore.net,53632,1369432814258.archivedHFileCleaner exiting 2013-05-24 22:01:02,554 INFO [Master:0;kiyo.gq1.ygridcore.net,53632,1369432814258.oldLogCleaner] master.ReplicationLogCleaner(154): Stopping replicationLogCleaner-0x13ed88e2cdf0015 2013-05-24 22:01:02,568 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392] regionserver.HRegionServer(941): stopping server kiyo.gq1.ygridcore.net,51523,1369432814392; zookeeper connection closed. 2013-05-24 22:01:02,568 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392] regionserver.HRegionServer(944): RegionServer:0;kiyo.gq1.ygridcore.net,51523,1369432814392 exiting 2013-05-24 22:01:02,569 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@5bcfa4b2] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(193): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@5bcfa4b2 2013-05-24 22:01:02,569 INFO [pool-1-thread-1] util.JVMClusterUtil(309): Shutdown of 1 master(s) and 2 regionserver(s) complete 2013-05-24 22:01:02,569 INFO [pool-1-thread-1] client.HConnectionManager$HConnectionImplementation(1640): Closing zookeeper sessionid=0x13ed88e2cdf0012 2013-05-24 22:01:02,620 INFO [Master:0;kiyo.gq1.ygridcore.net,53632,1369432814258.oldLogCleaner] master.ReplicationLogCleaner(158): Stopping org.apache.hadoop.hbase.replication.ReplicationStateImpl@2109a1be 2013-05-24 22:01:02,639 ERROR [Master:0;kiyo.gq1.ygridcore.net,53632,1369432814258.oldLogCleaner] client.HConnectionManager(362): Connection not found in the list, can't delete it (connection key=HConnectionKey{properties={hbase.rpc.timeout=60000, hbase.zookeeper.property.clientPort=51411, hbase.client.pause=1000, zookeeper.znode.parent=/2, hbase.client.retries.number=60, hbase.zookeeper.quorum=localhost}, username='hortonzy'}). May be the key was modified? 2013-05-24 22:01:02,659 INFO [Master:0;kiyo.gq1.ygridcore.net,53632,1369432814258] master.HMaster(587): HMaster main thread exiting 2013-05-24 22:01:02,660 WARN [pool-1-thread-1] datanode.DirectoryScanner(289): DirectoryScanner: shutdown has been called 2013-05-24 22:01:02,665 INFO [pool-1-thread-1] log.Slf4jLog(67): Stopped SelectChannelConnector@localhost:0 2013-05-24 22:01:02,769 WARN [DataNode: [file:/homes/hortonzy/trunk/hbase-server/target/test-data/f01230ff-021f-4338-8530-78663d37a44b/dfscluster_b4bc16e0-ae4c-45dd-be9c-abd12c3dc2b5/dfs/data/data3,file:/homes/hortonzy/trunk/hbase-server/target/test-data/f01230ff-021f-4338-8530-78663d37a44b/dfscluster_b4bc16e0-ae4c-45dd-be9c-abd12c3dc2b5/dfs/data/data4] heartbeating to localhost/127.0.0.1:54104] datanode.BPServiceActor(575): BPOfferService for Block pool BP-850081120-68.142.244.61-1369432810673 (storage id DS-386951416-68.142.244.61-33054-1369432813401) service to localhost/127.0.0.1:54104 interrupted 2013-05-24 22:01:02,769 WARN [DataNode: [file:/homes/hortonzy/trunk/hbase-server/target/test-data/f01230ff-021f-4338-8530-78663d37a44b/dfscluster_b4bc16e0-ae4c-45dd-be9c-abd12c3dc2b5/dfs/data/data3,file:/homes/hortonzy/trunk/hbase-server/target/test-data/f01230ff-021f-4338-8530-78663d37a44b/dfscluster_b4bc16e0-ae4c-45dd-be9c-abd12c3dc2b5/dfs/data/data4] heartbeating to localhost/127.0.0.1:54104] datanode.BPServiceActor(685): Ending block pool service for: Block pool BP-850081120-68.142.244.61-1369432810673 (storage id DS-386951416-68.142.244.61-33054-1369432813401) service to localhost/127.0.0.1:54104 2013-05-24 22:01:02,771 WARN [pool-1-thread-1] datanode.DirectoryScanner(289): DirectoryScanner: shutdown has been called 2013-05-24 22:01:02,774 INFO [pool-1-thread-1] log.Slf4jLog(67): Stopped SelectChannelConnector@localhost:0 2013-05-24 22:01:02,877 WARN [DataNode: [file:/homes/hortonzy/trunk/hbase-server/target/test-data/f01230ff-021f-4338-8530-78663d37a44b/dfscluster_b4bc16e0-ae4c-45dd-be9c-abd12c3dc2b5/dfs/data/data1,file:/homes/hortonzy/trunk/hbase-server/target/test-data/f01230ff-021f-4338-8530-78663d37a44b/dfscluster_b4bc16e0-ae4c-45dd-be9c-abd12c3dc2b5/dfs/data/data2] heartbeating to localhost/127.0.0.1:54104] datanode.BPServiceActor(575): BPOfferService for Block pool BP-850081120-68.142.244.61-1369432810673 (storage id DS-1949187577-68.142.244.61-52910-1369432813359) service to localhost/127.0.0.1:54104 interrupted 2013-05-24 22:01:02,877 WARN [DataNode: [file:/homes/hortonzy/trunk/hbase-server/target/test-data/f01230ff-021f-4338-8530-78663d37a44b/dfscluster_b4bc16e0-ae4c-45dd-be9c-abd12c3dc2b5/dfs/data/data1,file:/homes/hortonzy/trunk/hbase-server/target/test-data/f01230ff-021f-4338-8530-78663d37a44b/dfscluster_b4bc16e0-ae4c-45dd-be9c-abd12c3dc2b5/dfs/data/data2] heartbeating to localhost/127.0.0.1:54104] datanode.BPServiceActor(685): Ending block pool service for: Block pool BP-850081120-68.142.244.61-1369432810673 (storage id DS-1949187577-68.142.244.61-52910-1369432813359) service to localhost/127.0.0.1:54104 2013-05-24 22:01:03,234 WARN [org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@10f0f6ac] blockmanagement.BlockManager$ReplicationMonitor(3110): ReplicationMonitor thread received InterruptedException. java.lang.InterruptedException: sleep interrupted at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3108) at java.lang.Thread.run(Thread.java:662) 2013-05-24 22:01:03,235 WARN [org.apache.hadoop.hdfs.server.blockmanagement.DecommissionManager$Monitor@206c2ea3] blockmanagement.DecommissionManager$Monitor(78): Monitor interrupted: java.lang.InterruptedException: sleep interrupted 2013-05-24 22:01:03,246 INFO [pool-1-thread-1] log.Slf4jLog(67): Stopped SelectChannelConnector@localhost:0 2013-05-24 22:01:03,343 INFO [kiyo.gq1.ygridcore.net,39146,1369432814418Replication Statistics #0] regionserver.Replication$ReplicationStatisticsThread(291): Sink: age in ms of last applied edit: 28248, total replicated edits: 2115 2013-05-24 22:01:03,368 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(920): Minicluster is down 2013-05-24 22:01:03,368 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(911): Shutting down minicluster 2013-05-24 22:01:03,368 DEBUG [pool-1-thread-1] util.JVMClusterUtil(237): Shutting down HBase Cluster 2013-05-24 22:01:03,369 INFO [pool-1-thread-1] master.HMaster(2216): Cluster shutdown requested 2013-05-24 22:01:03,369 INFO [Master:0;kiyo.gq1.ygridcore.net,49101,1369432806564] master.ServerManager(448): Waiting on regionserver(s) to go down kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:01:03,369 INFO [kiyo.gq1.ygridcore.net,49101,1369432806564-CatalogJanitor] hbase.Chore(93): kiyo.gq1.ygridcore.net,49101,1369432806564-CatalogJanitor exiting 2013-05-24 22:01:03,369 INFO [kiyo.gq1.ygridcore.net,49101,1369432806564-ClusterStatusChore] hbase.Chore(93): kiyo.gq1.ygridcore.net,49101,1369432806564-ClusterStatusChore exiting 2013-05-24 22:01:03,369 INFO [kiyo.gq1.ygridcore.net,49101,1369432806564-BalancerChore] hbase.Chore(93): kiyo.gq1.ygridcore.net,49101,1369432806564-BalancerChore exiting 2013-05-24 22:01:03,393 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/running 2013-05-24 22:01:03,393 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:42690-0x13ed88e2cdf0006 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/running 2013-05-24 22:01:03,393 INFO [pool-1-thread-1] regionserver.HRegionServer(1639): STOPPED: Shutdown requested 2013-05-24 22:01:03,394 INFO [pool-1-thread-1] regionserver.HRegionServer(1639): STOPPED: Shutdown requested 2013-05-24 22:01:03,396 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911] regionserver.SplitLogWorker(585): Sending interrupt to stop the worker thread 2013-05-24 22:01:03,396 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,42690,1369432806911] regionserver.SplitLogWorker(267): SplitLogWorker interrupted while waiting for task, exiting: java.lang.InterruptedException 2013-05-24 22:01:03,396 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911] snapshot.RegionServerSnapshotManager(151): Stopping RegionServerSnapshotManager gracefully. 2013-05-24 22:01:03,396 INFO [Thread-151] regionserver.MemStoreFlusher$FlushHandler(267): Thread-151 exiting 2013-05-24 22:01:03,396 INFO [SplitLogWorker-kiyo.gq1.ygridcore.net,42690,1369432806911] regionserver.SplitLogWorker(199): SplitLogWorker kiyo.gq1.ygridcore.net,42690,1369432806911 exiting 2013-05-24 22:01:03,396 INFO [RS_OPEN_META-kiyo.gq1.ygridcore.net,42690,1369432806911-0MetaLogRoller] regionserver.LogRoller(119): LogRoller exiting. 2013-05-24 22:01:03,396 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.compactionChecker] hbase.Chore(93): RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.compactionChecker exiting 2013-05-24 22:01:03,396 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.logRoller] regionserver.LogRoller(119): LogRoller exiting. 2013-05-24 22:01:03,396 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(433): master:49101-0x13ed88e2cdf0004 Set watcher on znode that does not yet exist, /1/running 2013-05-24 22:01:03,398 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.ZKUtil(433): regionserver:42690-0x13ed88e2cdf0006 Set watcher on znode that does not yet exist, /1/running 2013-05-24 22:01:03,398 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] handler.CloseRegionHandler(125): Processing close of test,rrr,1369432820320.85815417b0e1742517fac3960fac86c1. 2013-05-24 22:01:03,398 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(927): Closing test,rrr,1369432820320.85815417b0e1742517fac3960fac86c1.: disabling compactions & flushes 2013-05-24 22:01:03,398 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] handler.CloseRegionHandler(125): Processing close of test,iii,1369432820320.992fe9e8d6a399ebc508d777668f74ad. 2013-05-24 22:01:03,398 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(949): Updates disabled for region test,rrr,1369432820320.85815417b0e1742517fac3960fac86c1. 2013-05-24 22:01:03,398 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] handler.CloseRegionHandler(125): Processing close of test,yyy,1369432820321.acf78311e603cb0d02309ae08763d196. 2013-05-24 22:01:03,399 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(1452): Started memstore flush for test,rrr,1369432820320.85815417b0e1742517fac3960fac86c1., current region memstore size 115.3 K 2013-05-24 22:01:03,399 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegion(927): Closing test,yyy,1369432820321.acf78311e603cb0d02309ae08763d196.: disabling compactions & flushes 2013-05-24 22:01:03,398 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegion(927): Closing test,iii,1369432820320.992fe9e8d6a399ebc508d777668f74ad.: disabling compactions & flushes 2013-05-24 22:01:03,399 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911] regionserver.HRegionServer(886): stopping server kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:01:03,399 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegion(949): Updates disabled for region test,yyy,1369432820321.acf78311e603cb0d02309ae08763d196. 2013-05-24 22:01:03,400 INFO [kiyo.gq1.ygridcore.net,51523,1369432814392Replication Statistics #0] regionserver.Replication$ReplicationStatisticsThread(291): Sink: age in ms of last applied edit: 21572, total replicated edits: 3593 2013-05-24 22:01:03,400 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegion(1452): Started memstore flush for test,yyy,1369432820321.acf78311e603cb0d02309ae08763d196., current region memstore size 115.3 K 2013-05-24 22:01:03,400 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911] catalog.CatalogTracker(208): Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@3736e10e 2013-05-24 22:01:03,399 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegion(949): Updates disabled for region test,iii,1369432820320.992fe9e8d6a399ebc508d777668f74ad. 2013-05-24 22:01:03,400 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911] snapshot.RegionServerSnapshotManager(151): Stopping RegionServerSnapshotManager gracefully. 2013-05-24 22:01:03,401 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegion(1452): Started memstore flush for test,iii,1369432820320.992fe9e8d6a399ebc508d777668f74ad., current region memstore size 115.3 K 2013-05-24 22:01:03,401 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911] regionserver.CompactSplitThread(318): Waiting for Split Thread to finish... 2013-05-24 22:01:03,402 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911] regionserver.CompactSplitThread(318): Waiting for Merge Thread to finish... 2013-05-24 22:01:03,402 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911] regionserver.CompactSplitThread(318): Waiting for Large Compaction Thread to finish... 2013-05-24 22:01:03,402 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911] regionserver.CompactSplitThread(318): Waiting for Small Compaction Thread to finish... 2013-05-24 22:01:03,402 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911] regionserver.HRegionServer(1048): Waiting on 27 regions to close 2013-05-24 22:01:03,402 DEBUG [RS_CLOSE_META-kiyo.gq1.ygridcore.net,42690,1369432806911-0] handler.CloseRegionHandler(125): Processing close of .META.,,1.1028785192 2013-05-24 22:01:03,403 DEBUG [RS_CLOSE_META-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(927): Closing .META.,,1.1028785192: disabling compactions & flushes 2013-05-24 22:01:03,403 DEBUG [RS_CLOSE_META-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(949): Updates disabled for region .META.,,1.1028785192 2013-05-24 22:01:03,403 DEBUG [RS_CLOSE_META-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(1452): Started memstore flush for .META.,,1.1028785192, current region memstore size 34.7 K 2013-05-24 22:01:03,439 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(388): Source exiting 2 2013-05-24 22:01:03,488 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.periodicFlusher] hbase.Chore(93): RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.periodicFlusher exiting 2013-05-24 22:01:03,490 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.leaseChecker] regionserver.Leases(124): RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.leaseChecker closing leases 2013-05-24 22:01:03,491 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.leaseChecker] regionserver.Leases(131): RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.leaseChecker closed leases 2013-05-24 22:01:03,704 INFO [IPC Server handler 7 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53333 is added to blk_2832164994964885218_1047{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:59993|RBW], ReplicaUnderConstruction[127.0.0.1:53333|RBW]]} size 0 2013-05-24 22:01:03,705 INFO [IPC Server handler 2 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59993 is added to blk_2832164994964885218_1047{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:59993|RBW], ReplicaUnderConstruction[127.0.0.1:53333|RBW]]} size 0 2013-05-24 22:01:03,706 INFO [IPC Server handler 8 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53333 is added to blk_2860850854083833178_1048{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:53333|RBW], ReplicaUnderConstruction[127.0.0.1:59993|RBW]]} size 21685 2013-05-24 22:01:03,706 INFO [IPC Server handler 5 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59993 is added to blk_2860850854083833178_1048 size 21685 2013-05-24 22:01:03,710 INFO [IPC Server handler 4 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53333 is added to blk_-6733238160818406181_1049{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:59993|RBW], ReplicaUnderConstruction[127.0.0.1:53333|RBW]]} size 0 2013-05-24 22:01:03,711 INFO [IPC Server handler 7 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59993 is added to blk_-6733238160818406181_1049{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:59993|RBW], ReplicaUnderConstruction[127.0.0.1:53333|RBW]]} size 0 2013-05-24 22:01:03,745 INFO [RS_CLOSE_META-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.DefaultStoreFlusher(88): Flushed, sequenceid=42, memsize=34.7 K, hasBloomFilter=false, into tmp file hdfs://localhost:54096/user/hortonzy/hbase/.META./1028785192/.tmp/13e983293dba41afa1281605fea7f774 2013-05-24 22:01:03,750 INFO [IPC Server handler 4 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59993 is added to blk_-935616985890139846_1050{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:53333|RBW], ReplicaUnderConstruction[127.0.0.1:59993|RBW]]} size 0 2013-05-24 22:01:03,751 INFO [IPC Server handler 3 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53333 is added to blk_-935616985890139846_1050 size 21685 2013-05-24 22:01:03,758 DEBUG [RS_CLOSE_META-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegionFileSystem(338): Committing store file hdfs://localhost:54096/user/hortonzy/hbase/.META./1028785192/.tmp/13e983293dba41afa1281605fea7f774 as hdfs://localhost:54096/user/hortonzy/hbase/.META./1028785192/info/13e983293dba41afa1281605fea7f774 2013-05-24 22:01:03,770 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.DefaultStoreFlusher(88): Flushed, sequenceid=3781, memsize=115.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:54096/user/hortonzy/hbase/test/85815417b0e1742517fac3960fac86c1/.tmp/162672bb870b4d9c8a06fdea21cef9b6 2013-05-24 22:01:03,779 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegionFileSystem(338): Committing store file hdfs://localhost:54096/user/hortonzy/hbase/test/85815417b0e1742517fac3960fac86c1/.tmp/162672bb870b4d9c8a06fdea21cef9b6 as hdfs://localhost:54096/user/hortonzy/hbase/test/85815417b0e1742517fac3960fac86c1/f/162672bb870b4d9c8a06fdea21cef9b6 2013-05-24 22:01:03,905 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.DefaultStoreFlusher(88): Flushed, sequenceid=3782, memsize=115.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:54096/user/hortonzy/hbase/test/acf78311e603cb0d02309ae08763d196/.tmp/32815363ee8f4fd39a592d9304611a33 2013-05-24 22:01:03,912 INFO [RS_CLOSE_META-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HStore(761): Added hdfs://localhost:54096/user/hortonzy/hbase/.META./1028785192/info/13e983293dba41afa1281605fea7f774, entries=143, sequenceid=42, filesize=16.4 K 2013-05-24 22:01:03,912 INFO [RS_CLOSE_META-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(1596): Finished memstore flush of ~34.7 K/35536, currentsize=0/0 for region .META.,,1.1028785192 in 509ms, sequenceid=42, compaction requested=false 2013-05-24 22:01:03,915 INFO [StoreCloserThread-.META.,,1.1028785192-1] regionserver.HStore(663): Closed info 2013-05-24 22:01:03,915 INFO [RS_CLOSE_META-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(1006): Closed .META.,,1.1028785192 2013-05-24 22:01:03,915 DEBUG [RS_CLOSE_META-kiyo.gq1.ygridcore.net,42690,1369432806911-0] handler.CloseRegionHandler(177): Closed region .META.,,1.1028785192 2013-05-24 22:01:03,952 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegionFileSystem(338): Committing store file hdfs://localhost:54096/user/hortonzy/hbase/test/acf78311e603cb0d02309ae08763d196/.tmp/32815363ee8f4fd39a592d9304611a33 as hdfs://localhost:54096/user/hortonzy/hbase/test/acf78311e603cb0d02309ae08763d196/f/32815363ee8f4fd39a592d9304611a33 2013-05-24 22:01:03,953 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HStore(761): Added hdfs://localhost:54096/user/hortonzy/hbase/test/85815417b0e1742517fac3960fac86c1/f/162672bb870b4d9c8a06fdea21cef9b6, entries=703, sequenceid=3781, filesize=21.2 K 2013-05-24 22:01:03,953 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(1596): Finished memstore flush of ~115.3 K/118104, currentsize=0/0 for region test,rrr,1369432820320.85815417b0e1742517fac3960fac86c1. in 555ms, sequenceid=3781, compaction requested=false 2013-05-24 22:01:03,955 INFO [StoreCloserThread-test,rrr,1369432820320.85815417b0e1742517fac3960fac86c1.-1] regionserver.HStore(663): Closed f 2013-05-24 22:01:03,955 INFO [StoreCloserThread-test,rrr,1369432820320.85815417b0e1742517fac3960fac86c1.-1] regionserver.HStore(663): Closed norep 2013-05-24 22:01:03,955 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(1006): Closed test,rrr,1369432820320.85815417b0e1742517fac3960fac86c1. 2013-05-24 22:01:03,955 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] handler.CloseRegionHandler(177): Closed region test,rrr,1369432820320.85815417b0e1742517fac3960fac86c1. 2013-05-24 22:01:03,955 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] handler.CloseRegionHandler(125): Processing close of test,kkk,1369432820320.1ce9aafaaa422cef4ce0e1895111c733. 2013-05-24 22:01:03,955 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(927): Closing test,kkk,1369432820320.1ce9aafaaa422cef4ce0e1895111c733.: disabling compactions & flushes 2013-05-24 22:01:03,955 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(949): Updates disabled for region test,kkk,1369432820320.1ce9aafaaa422cef4ce0e1895111c733. 2013-05-24 22:01:03,956 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(1452): Started memstore flush for test,kkk,1369432820320.1ce9aafaaa422cef4ce0e1895111c733., current region memstore size 115.3 K 2013-05-24 22:01:04,112 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HStore(761): Added hdfs://localhost:54096/user/hortonzy/hbase/test/acf78311e603cb0d02309ae08763d196/f/32815363ee8f4fd39a592d9304611a33, entries=703, sequenceid=3782, filesize=21.2 K 2013-05-24 22:01:04,112 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegion(1596): Finished memstore flush of ~115.3 K/118104, currentsize=0/0 for region test,yyy,1369432820321.acf78311e603cb0d02309ae08763d196. in 712ms, sequenceid=3782, compaction requested=false 2013-05-24 22:01:04,114 INFO [StoreCloserThread-test,yyy,1369432820321.acf78311e603cb0d02309ae08763d196.-1] regionserver.HStore(663): Closed f 2013-05-24 22:01:04,114 INFO [StoreCloserThread-test,yyy,1369432820321.acf78311e603cb0d02309ae08763d196.-1] regionserver.HStore(663): Closed norep 2013-05-24 22:01:04,115 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegion(1006): Closed test,yyy,1369432820321.acf78311e603cb0d02309ae08763d196. 2013-05-24 22:01:04,115 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] handler.CloseRegionHandler(177): Closed region test,yyy,1369432820321.acf78311e603cb0d02309ae08763d196. 2013-05-24 22:01:04,115 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] handler.CloseRegionHandler(125): Processing close of test,ooo,1369432820320.0d6df81a73ad3b2a79add80716da62ac. 2013-05-24 22:01:04,115 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegion(927): Closing test,ooo,1369432820320.0d6df81a73ad3b2a79add80716da62ac.: disabling compactions & flushes 2013-05-24 22:01:04,115 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegion(949): Updates disabled for region test,ooo,1369432820320.0d6df81a73ad3b2a79add80716da62ac. 2013-05-24 22:01:04,115 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegion(1452): Started memstore flush for test,ooo,1369432820320.0d6df81a73ad3b2a79add80716da62ac., current region memstore size 115.3 K 2013-05-24 22:01:04,260 INFO [kiyo.gq1.ygridcore.net,49101,1369432806564.splitLogManagerTimeoutMonitor] hbase.Chore(93): kiyo.gq1.ygridcore.net,49101,1369432806564.splitLogManagerTimeoutMonitor exiting 2013-05-24 22:01:04,293 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.DefaultStoreFlusher(88): Flushed, sequenceid=3783, memsize=115.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:54096/user/hortonzy/hbase/test/992fe9e8d6a399ebc508d777668f74ad/.tmp/b2eac4b7c25746dcb763ae5580526556 2013-05-24 22:01:04,301 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegionFileSystem(338): Committing store file hdfs://localhost:54096/user/hortonzy/hbase/test/992fe9e8d6a399ebc508d777668f74ad/.tmp/b2eac4b7c25746dcb763ae5580526556 as hdfs://localhost:54096/user/hortonzy/hbase/test/992fe9e8d6a399ebc508d777668f74ad/f/b2eac4b7c25746dcb763ae5580526556 2013-05-24 22:01:04,371 INFO [Master:0;kiyo.gq1.ygridcore.net,49101,1369432806564] master.ServerManager(448): Waiting on regionserver(s) to go down kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:01:04,403 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911] regionserver.HRegionServer(1048): Waiting on 24 regions to close 2013-05-24 22:01:04,409 INFO [IPC Server handler 9 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53333 is added to blk_-5869906588369293188_1051{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:59993|RBW], ReplicaUnderConstruction[127.0.0.1:53333|RBW]]} size 0 2013-05-24 22:01:04,409 INFO [IPC Server handler 6 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59993 is added to blk_-5869906588369293188_1051 size 21685 2013-05-24 22:01:04,572 INFO [IPC Server handler 9 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53333 is added to blk_5102666628319673696_1052{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:59993|RBW], ReplicaUnderConstruction[127.0.0.1:53333|RBW]]} size 0 2013-05-24 22:01:04,572 INFO [IPC Server handler 3 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59993 is added to blk_5102666628319673696_1052 size 21685 2013-05-24 22:01:04,577 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HStore(761): Added hdfs://localhost:54096/user/hortonzy/hbase/test/992fe9e8d6a399ebc508d777668f74ad/f/b2eac4b7c25746dcb763ae5580526556, entries=703, sequenceid=3783, filesize=21.2 K 2013-05-24 22:01:04,577 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegion(1596): Finished memstore flush of ~115.3 K/118104, currentsize=0/0 for region test,iii,1369432820320.992fe9e8d6a399ebc508d777668f74ad. in 1176ms, sequenceid=3783, compaction requested=false 2013-05-24 22:01:04,579 INFO [StoreCloserThread-test,iii,1369432820320.992fe9e8d6a399ebc508d777668f74ad.-1] regionserver.HStore(663): Closed f 2013-05-24 22:01:04,579 INFO [StoreCloserThread-test,iii,1369432820320.992fe9e8d6a399ebc508d777668f74ad.-1] regionserver.HStore(663): Closed norep 2013-05-24 22:01:04,580 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegion(1006): Closed test,iii,1369432820320.992fe9e8d6a399ebc508d777668f74ad. 2013-05-24 22:01:04,580 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] handler.CloseRegionHandler(177): Closed region test,iii,1369432820320.992fe9e8d6a399ebc508d777668f74ad. 2013-05-24 22:01:04,580 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] handler.CloseRegionHandler(125): Processing close of test,zzz,1369432820321.0b4d70d096cb6cbce3376acd563c77dd. 2013-05-24 22:01:04,580 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegion(927): Closing test,zzz,1369432820321.0b4d70d096cb6cbce3376acd563c77dd.: disabling compactions & flushes 2013-05-24 22:01:04,580 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegion(949): Updates disabled for region test,zzz,1369432820321.0b4d70d096cb6cbce3376acd563c77dd. 2013-05-24 22:01:04,580 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegion(1452): Started memstore flush for test,zzz,1369432820321.0b4d70d096cb6cbce3376acd563c77dd., current region memstore size 168 2013-05-24 22:01:04,791 INFO [kiyo.gq1.ygridcore.net,42690,1369432806911Replication Statistics #0] regionserver.Replication$ReplicationStatisticsThread(291): Normal source for cluster 2: Total replicated edits: 2667, currently replicating from: hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432840428 at position: N/A 2013-05-24 22:01:04,791 INFO [kiyo.gq1.ygridcore.net,49701,1369432806997Replication Statistics #0] regionserver.Replication$ReplicationStatisticsThread(291): Normal source for cluster 2: Total replicated edits: 1629, currently replicating from: hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432832342 at position: N/A 2013-05-24 22:01:04,912 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.DefaultStoreFlusher(88): Flushed, sequenceid=3784, memsize=115.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:54096/user/hortonzy/hbase/test/1ce9aafaaa422cef4ce0e1895111c733/.tmp/e643591567d040da89214d78d629b575 2013-05-24 22:01:04,920 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegionFileSystem(338): Committing store file hdfs://localhost:54096/user/hortonzy/hbase/test/1ce9aafaaa422cef4ce0e1895111c733/.tmp/e643591567d040da89214d78d629b575 as hdfs://localhost:54096/user/hortonzy/hbase/test/1ce9aafaaa422cef4ce0e1895111c733/f/e643591567d040da89214d78d629b575 2013-05-24 22:01:04,969 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.DefaultStoreFlusher(88): Flushed, sequenceid=3785, memsize=115.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:54096/user/hortonzy/hbase/test/0d6df81a73ad3b2a79add80716da62ac/.tmp/0cf563c53cbe4cf4b6341fcadaad71fd 2013-05-24 22:01:04,976 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegionFileSystem(338): Committing store file hdfs://localhost:54096/user/hortonzy/hbase/test/0d6df81a73ad3b2a79add80716da62ac/.tmp/0cf563c53cbe4cf4b6341fcadaad71fd as hdfs://localhost:54096/user/hortonzy/hbase/test/0d6df81a73ad3b2a79add80716da62ac/f/0cf563c53cbe4cf4b6341fcadaad71fd 2013-05-24 22:01:05,067 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HStore(761): Added hdfs://localhost:54096/user/hortonzy/hbase/test/1ce9aafaaa422cef4ce0e1895111c733/f/e643591567d040da89214d78d629b575, entries=703, sequenceid=3784, filesize=21.2 K 2013-05-24 22:01:05,068 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(1596): Finished memstore flush of ~115.3 K/118104, currentsize=0/0 for region test,kkk,1369432820320.1ce9aafaaa422cef4ce0e1895111c733. in 1113ms, sequenceid=3784, compaction requested=false 2013-05-24 22:01:05,070 INFO [StoreCloserThread-test,kkk,1369432820320.1ce9aafaaa422cef4ce0e1895111c733.-1] regionserver.HStore(663): Closed f 2013-05-24 22:01:05,070 INFO [StoreCloserThread-test,kkk,1369432820320.1ce9aafaaa422cef4ce0e1895111c733.-1] regionserver.HStore(663): Closed norep 2013-05-24 22:01:05,071 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(1006): Closed test,kkk,1369432820320.1ce9aafaaa422cef4ce0e1895111c733. 2013-05-24 22:01:05,071 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] handler.CloseRegionHandler(177): Closed region test,kkk,1369432820320.1ce9aafaaa422cef4ce0e1895111c733. 2013-05-24 22:01:05,071 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] handler.CloseRegionHandler(125): Processing close of test,fff,1369432820319.f852f52d87b0b7acd213f5b49074dde6. 2013-05-24 22:01:05,071 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(927): Closing test,fff,1369432820319.f852f52d87b0b7acd213f5b49074dde6.: disabling compactions & flushes 2013-05-24 22:01:05,071 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(949): Updates disabled for region test,fff,1369432820319.f852f52d87b0b7acd213f5b49074dde6. 2013-05-24 22:01:05,071 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(1452): Started memstore flush for test,fff,1369432820319.f852f52d87b0b7acd213f5b49074dde6., current region memstore size 115.3 K 2013-05-24 22:01:05,338 INFO [IPC Server handler 8 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53333 is added to blk_-4872780623939172674_1053{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:59993|RBW], ReplicaUnderConstruction[127.0.0.1:53333|RBW]]} size 0 2013-05-24 22:01:05,339 INFO [IPC Server handler 3 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59993 is added to blk_-4872780623939172674_1053 size 1003 2013-05-24 22:01:05,344 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HStore(761): Added hdfs://localhost:54096/user/hortonzy/hbase/test/0d6df81a73ad3b2a79add80716da62ac/f/0cf563c53cbe4cf4b6341fcadaad71fd, entries=703, sequenceid=3785, filesize=21.2 K 2013-05-24 22:01:05,344 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegion(1596): Finished memstore flush of ~115.3 K/118104, currentsize=0/0 for region test,ooo,1369432820320.0d6df81a73ad3b2a79add80716da62ac. in 1229ms, sequenceid=3785, compaction requested=false 2013-05-24 22:01:05,346 INFO [StoreCloserThread-test,ooo,1369432820320.0d6df81a73ad3b2a79add80716da62ac.-1] regionserver.HStore(663): Closed f 2013-05-24 22:01:05,347 INFO [StoreCloserThread-test,ooo,1369432820320.0d6df81a73ad3b2a79add80716da62ac.-1] regionserver.HStore(663): Closed norep 2013-05-24 22:01:05,347 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegion(1006): Closed test,ooo,1369432820320.0d6df81a73ad3b2a79add80716da62ac. 2013-05-24 22:01:05,347 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] handler.CloseRegionHandler(177): Closed region test,ooo,1369432820320.0d6df81a73ad3b2a79add80716da62ac. 2013-05-24 22:01:05,347 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] handler.CloseRegionHandler(125): Processing close of test,uuu,1369432820320.dba8b67ade91438cad8a5209736330e6. 2013-05-24 22:01:05,347 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegion(927): Closing test,uuu,1369432820320.dba8b67ade91438cad8a5209736330e6.: disabling compactions & flushes 2013-05-24 22:01:05,348 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegion(949): Updates disabled for region test,uuu,1369432820320.dba8b67ade91438cad8a5209736330e6. 2013-05-24 22:01:05,348 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegion(1452): Started memstore flush for test,uuu,1369432820320.dba8b67ade91438cad8a5209736330e6., current region memstore size 115.3 K 2013-05-24 22:01:05,372 INFO [Master:0;kiyo.gq1.ygridcore.net,49101,1369432806564] master.ServerManager(448): Waiting on regionserver(s) to go down kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:01:05,404 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911] regionserver.HRegionServer(1048): Waiting on 21 regions to close 2013-05-24 22:01:05,546 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.DefaultStoreFlusher(88): Flushed, sequenceid=3786, memsize=168, hasBloomFilter=true, into tmp file hdfs://localhost:54096/user/hortonzy/hbase/test/0b4d70d096cb6cbce3376acd563c77dd/.tmp/03a6006f08034f3a9699c28b31a6b663 2013-05-24 22:01:05,554 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegionFileSystem(338): Committing store file hdfs://localhost:54096/user/hortonzy/hbase/test/0b4d70d096cb6cbce3376acd563c77dd/.tmp/03a6006f08034f3a9699c28b31a6b663 as hdfs://localhost:54096/user/hortonzy/hbase/test/0b4d70d096cb6cbce3376acd563c77dd/f/03a6006f08034f3a9699c28b31a6b663 2013-05-24 22:01:05,658 INFO [IPC Server handler 8 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59993 is added to blk_3331264006866116977_1054{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:53333|RBW], ReplicaUnderConstruction[127.0.0.1:59993|RBW]]} size 0 2013-05-24 22:01:05,659 INFO [IPC Server handler 1 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53333 is added to blk_3331264006866116977_1054 size 21685 2013-05-24 22:01:05,755 INFO [IPC Server handler 1 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53333 is added to blk_128427888200092184_1055{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:59993|RBW], ReplicaUnderConstruction[127.0.0.1:53333|RBW]]} size 0 2013-05-24 22:01:05,756 INFO [IPC Server handler 7 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59993 is added to blk_128427888200092184_1055 size 21685 2013-05-24 22:01:05,759 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HStore(761): Added hdfs://localhost:54096/user/hortonzy/hbase/test/0b4d70d096cb6cbce3376acd563c77dd/f/03a6006f08034f3a9699c28b31a6b663, entries=1, sequenceid=3786, filesize=1003 2013-05-24 22:01:05,759 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegion(1596): Finished memstore flush of ~168/168, currentsize=0/0 for region test,zzz,1369432820321.0b4d70d096cb6cbce3376acd563c77dd. in 1179ms, sequenceid=3786, compaction requested=false 2013-05-24 22:01:05,761 INFO [StoreCloserThread-test,zzz,1369432820321.0b4d70d096cb6cbce3376acd563c77dd.-1] regionserver.HStore(663): Closed f 2013-05-24 22:01:05,761 INFO [StoreCloserThread-test,zzz,1369432820321.0b4d70d096cb6cbce3376acd563c77dd.-1] regionserver.HStore(663): Closed norep 2013-05-24 22:01:05,761 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegion(1006): Closed test,zzz,1369432820321.0b4d70d096cb6cbce3376acd563c77dd. 2013-05-24 22:01:05,761 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] handler.CloseRegionHandler(177): Closed region test,zzz,1369432820321.0b4d70d096cb6cbce3376acd563c77dd. 2013-05-24 22:01:05,762 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] handler.CloseRegionHandler(125): Processing close of test,ggg,1369432820319.4b3ff3efde12c5afe1220e81d155ac73. 2013-05-24 22:01:05,762 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegion(927): Closing test,ggg,1369432820319.4b3ff3efde12c5afe1220e81d155ac73.: disabling compactions & flushes 2013-05-24 22:01:05,762 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegion(949): Updates disabled for region test,ggg,1369432820319.4b3ff3efde12c5afe1220e81d155ac73. 2013-05-24 22:01:05,762 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegion(1452): Started memstore flush for test,ggg,1369432820319.4b3ff3efde12c5afe1220e81d155ac73., current region memstore size 115.3 K 2013-05-24 22:01:05,776 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.DefaultStoreFlusher(88): Flushed, sequenceid=3787, memsize=115.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:54096/user/hortonzy/hbase/test/f852f52d87b0b7acd213f5b49074dde6/.tmp/bc6891cd8c8e4d9a8c66373abf314446 2013-05-24 22:01:05,783 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegionFileSystem(338): Committing store file hdfs://localhost:54096/user/hortonzy/hbase/test/f852f52d87b0b7acd213f5b49074dde6/.tmp/bc6891cd8c8e4d9a8c66373abf314446 as hdfs://localhost:54096/user/hortonzy/hbase/test/f852f52d87b0b7acd213f5b49074dde6/f/bc6891cd8c8e4d9a8c66373abf314446 2013-05-24 22:01:05,908 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.DefaultStoreFlusher(88): Flushed, sequenceid=3788, memsize=115.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:54096/user/hortonzy/hbase/test/dba8b67ade91438cad8a5209736330e6/.tmp/4e6ca9a6d0f5464ebe2ac2193e38b79a 2013-05-24 22:01:05,917 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegionFileSystem(338): Committing store file hdfs://localhost:54096/user/hortonzy/hbase/test/dba8b67ade91438cad8a5209736330e6/.tmp/4e6ca9a6d0f5464ebe2ac2193e38b79a as hdfs://localhost:54096/user/hortonzy/hbase/test/dba8b67ade91438cad8a5209736330e6/f/4e6ca9a6d0f5464ebe2ac2193e38b79a 2013-05-24 22:01:05,959 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HStore(761): Added hdfs://localhost:54096/user/hortonzy/hbase/test/f852f52d87b0b7acd213f5b49074dde6/f/bc6891cd8c8e4d9a8c66373abf314446, entries=703, sequenceid=3787, filesize=21.2 K 2013-05-24 22:01:05,960 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(1596): Finished memstore flush of ~115.3 K/118104, currentsize=0/0 for region test,fff,1369432820319.f852f52d87b0b7acd213f5b49074dde6. in 889ms, sequenceid=3787, compaction requested=false 2013-05-24 22:01:05,962 INFO [StoreCloserThread-test,fff,1369432820319.f852f52d87b0b7acd213f5b49074dde6.-1] regionserver.HStore(663): Closed f 2013-05-24 22:01:05,962 INFO [StoreCloserThread-test,fff,1369432820319.f852f52d87b0b7acd213f5b49074dde6.-1] regionserver.HStore(663): Closed norep 2013-05-24 22:01:05,962 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(1006): Closed test,fff,1369432820319.f852f52d87b0b7acd213f5b49074dde6. 2013-05-24 22:01:05,963 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] handler.CloseRegionHandler(177): Closed region test,fff,1369432820319.f852f52d87b0b7acd213f5b49074dde6. 2013-05-24 22:01:05,963 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] handler.CloseRegionHandler(125): Processing close of test,sss,1369432820320.872960a61a4ceeb9f64577d8b7cb0ddf. 2013-05-24 22:01:05,963 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(927): Closing test,sss,1369432820320.872960a61a4ceeb9f64577d8b7cb0ddf.: disabling compactions & flushes 2013-05-24 22:01:05,963 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(949): Updates disabled for region test,sss,1369432820320.872960a61a4ceeb9f64577d8b7cb0ddf. 2013-05-24 22:01:05,963 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(1452): Started memstore flush for test,sss,1369432820320.872960a61a4ceeb9f64577d8b7cb0ddf., current region memstore size 115.3 K 2013-05-24 22:01:06,005 INFO [IPC Server handler 5 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59993 is added to blk_4529517233622179859_1056{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:53333|RBW], ReplicaUnderConstruction[127.0.0.1:59993|RBW]]} size 0 2013-05-24 22:01:06,005 INFO [IPC Server handler 0 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53333 is added to blk_4529517233622179859_1056 size 21685 2013-05-24 22:01:06,009 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HStore(761): Added hdfs://localhost:54096/user/hortonzy/hbase/test/dba8b67ade91438cad8a5209736330e6/f/4e6ca9a6d0f5464ebe2ac2193e38b79a, entries=703, sequenceid=3788, filesize=21.2 K 2013-05-24 22:01:06,009 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegion(1596): Finished memstore flush of ~115.3 K/118104, currentsize=0/0 for region test,uuu,1369432820320.dba8b67ade91438cad8a5209736330e6. in 661ms, sequenceid=3788, compaction requested=false 2013-05-24 22:01:06,011 INFO [StoreCloserThread-test,uuu,1369432820320.dba8b67ade91438cad8a5209736330e6.-1] regionserver.HStore(663): Closed f 2013-05-24 22:01:06,012 INFO [StoreCloserThread-test,uuu,1369432820320.dba8b67ade91438cad8a5209736330e6.-1] regionserver.HStore(663): Closed norep 2013-05-24 22:01:06,012 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegion(1006): Closed test,uuu,1369432820320.dba8b67ade91438cad8a5209736330e6. 2013-05-24 22:01:06,012 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] handler.CloseRegionHandler(177): Closed region test,uuu,1369432820320.dba8b67ade91438cad8a5209736330e6. 2013-05-24 22:01:06,012 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] handler.CloseRegionHandler(125): Processing close of test,,1369432820315.a88e8c9e73f832adb3635d6996d9e72d. 2013-05-24 22:01:06,012 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegion(927): Closing test,,1369432820315.a88e8c9e73f832adb3635d6996d9e72d.: disabling compactions & flushes 2013-05-24 22:01:06,012 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegion(949): Updates disabled for region test,,1369432820315.a88e8c9e73f832adb3635d6996d9e72d. 2013-05-24 22:01:06,013 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegion(1452): Started memstore flush for test,,1369432820315.a88e8c9e73f832adb3635d6996d9e72d., current region memstore size 115.3 K 2013-05-24 22:01:06,045 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.DefaultStoreFlusher(88): Flushed, sequenceid=3789, memsize=115.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:54096/user/hortonzy/hbase/test/4b3ff3efde12c5afe1220e81d155ac73/.tmp/9903a8a6be90468290a34b99a5ebf7c4 2013-05-24 22:01:06,052 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegionFileSystem(338): Committing store file hdfs://localhost:54096/user/hortonzy/hbase/test/4b3ff3efde12c5afe1220e81d155ac73/.tmp/9903a8a6be90468290a34b99a5ebf7c4 as hdfs://localhost:54096/user/hortonzy/hbase/test/4b3ff3efde12c5afe1220e81d155ac73/f/9903a8a6be90468290a34b99a5ebf7c4 2013-05-24 22:01:06,086 INFO [IPC Server handler 5 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53333 is added to blk_-5865712870651664591_1057{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:59993|RBW], ReplicaUnderConstruction[127.0.0.1:53333|RBW]]} size 0 2013-05-24 22:01:06,087 INFO [IPC Server handler 3 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59993 is added to blk_-5865712870651664591_1057 size 21685 2013-05-24 22:01:06,149 INFO [IPC Server handler 5 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53333 is added to blk_-3848850836528531098_1058{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:59993|RBW], ReplicaUnderConstruction[127.0.0.1:53333|RBW]]} size 0 2013-05-24 22:01:06,150 INFO [IPC Server handler 8 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59993 is added to blk_-3848850836528531098_1058 size 21685 2013-05-24 22:01:06,154 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HStore(761): Added hdfs://localhost:54096/user/hortonzy/hbase/test/4b3ff3efde12c5afe1220e81d155ac73/f/9903a8a6be90468290a34b99a5ebf7c4, entries=703, sequenceid=3789, filesize=21.2 K 2013-05-24 22:01:06,154 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegion(1596): Finished memstore flush of ~115.3 K/118104, currentsize=0/0 for region test,ggg,1369432820319.4b3ff3efde12c5afe1220e81d155ac73. in 392ms, sequenceid=3789, compaction requested=false 2013-05-24 22:01:06,156 INFO [StoreCloserThread-test,ggg,1369432820319.4b3ff3efde12c5afe1220e81d155ac73.-1] regionserver.HStore(663): Closed f 2013-05-24 22:01:06,156 INFO [StoreCloserThread-test,ggg,1369432820319.4b3ff3efde12c5afe1220e81d155ac73.-1] regionserver.HStore(663): Closed norep 2013-05-24 22:01:06,156 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegion(1006): Closed test,ggg,1369432820319.4b3ff3efde12c5afe1220e81d155ac73. 2013-05-24 22:01:06,156 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] handler.CloseRegionHandler(177): Closed region test,ggg,1369432820319.4b3ff3efde12c5afe1220e81d155ac73. 2013-05-24 22:01:06,156 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] handler.CloseRegionHandler(125): Processing close of test,www,1369432820320.643c76d94d70be44d35b5a12853bb852. 2013-05-24 22:01:06,156 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegion(927): Closing test,www,1369432820320.643c76d94d70be44d35b5a12853bb852.: disabling compactions & flushes 2013-05-24 22:01:06,156 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegion(949): Updates disabled for region test,www,1369432820320.643c76d94d70be44d35b5a12853bb852. 2013-05-24 22:01:06,156 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegion(1452): Started memstore flush for test,www,1369432820320.643c76d94d70be44d35b5a12853bb852., current region memstore size 115.3 K 2013-05-24 22:01:06,179 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.DefaultStoreFlusher(88): Flushed, sequenceid=3790, memsize=115.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:54096/user/hortonzy/hbase/test/872960a61a4ceeb9f64577d8b7cb0ddf/.tmp/8e3d11285f8f4a93871aa35b04d1ec6b 2013-05-24 22:01:06,187 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegionFileSystem(338): Committing store file hdfs://localhost:54096/user/hortonzy/hbase/test/872960a61a4ceeb9f64577d8b7cb0ddf/.tmp/8e3d11285f8f4a93871aa35b04d1ec6b as hdfs://localhost:54096/user/hortonzy/hbase/test/872960a61a4ceeb9f64577d8b7cb0ddf/f/8e3d11285f8f4a93871aa35b04d1ec6b 2013-05-24 22:01:06,278 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.DefaultStoreFlusher(88): Flushed, sequenceid=3791, memsize=115.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:54096/user/hortonzy/hbase/test/a88e8c9e73f832adb3635d6996d9e72d/.tmp/cf3fb5911142449890bc91d65bfacf11 2013-05-24 22:01:06,285 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegionFileSystem(338): Committing store file hdfs://localhost:54096/user/hortonzy/hbase/test/a88e8c9e73f832adb3635d6996d9e72d/.tmp/cf3fb5911142449890bc91d65bfacf11 as hdfs://localhost:54096/user/hortonzy/hbase/test/a88e8c9e73f832adb3635d6996d9e72d/f/cf3fb5911142449890bc91d65bfacf11 2013-05-24 22:01:06,332 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HStore(761): Added hdfs://localhost:54096/user/hortonzy/hbase/test/872960a61a4ceeb9f64577d8b7cb0ddf/f/8e3d11285f8f4a93871aa35b04d1ec6b, entries=703, sequenceid=3790, filesize=21.2 K 2013-05-24 22:01:06,332 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(1596): Finished memstore flush of ~115.3 K/118104, currentsize=0/0 for region test,sss,1369432820320.872960a61a4ceeb9f64577d8b7cb0ddf. in 369ms, sequenceid=3790, compaction requested=false 2013-05-24 22:01:06,334 INFO [StoreCloserThread-test,sss,1369432820320.872960a61a4ceeb9f64577d8b7cb0ddf.-1] regionserver.HStore(663): Closed f 2013-05-24 22:01:06,334 INFO [StoreCloserThread-test,sss,1369432820320.872960a61a4ceeb9f64577d8b7cb0ddf.-1] regionserver.HStore(663): Closed norep 2013-05-24 22:01:06,335 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(1006): Closed test,sss,1369432820320.872960a61a4ceeb9f64577d8b7cb0ddf. 2013-05-24 22:01:06,335 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] handler.CloseRegionHandler(177): Closed region test,sss,1369432820320.872960a61a4ceeb9f64577d8b7cb0ddf. 2013-05-24 22:01:06,335 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] handler.CloseRegionHandler(125): Processing close of test,ccc,1369432820319.117d9d6b052965f7196396ea21f07a61. 2013-05-24 22:01:06,335 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(927): Closing test,ccc,1369432820319.117d9d6b052965f7196396ea21f07a61.: disabling compactions & flushes 2013-05-24 22:01:06,335 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(949): Updates disabled for region test,ccc,1369432820319.117d9d6b052965f7196396ea21f07a61. 2013-05-24 22:01:06,335 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(1452): Started memstore flush for test,ccc,1369432820319.117d9d6b052965f7196396ea21f07a61., current region memstore size 115.3 K 2013-05-24 22:01:06,374 INFO [Master:0;kiyo.gq1.ygridcore.net,49101,1369432806564] master.ServerManager(448): Waiting on regionserver(s) to go down kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:01:06,405 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911] regionserver.HRegionServer(1048): Waiting on 16 regions to close 2013-05-24 22:01:06,412 INFO [IPC Server handler 0 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59993 is added to blk_7946535798164252757_1059{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:53333|RBW], ReplicaUnderConstruction[127.0.0.1:59993|RBW]]} size 0 2013-05-24 22:01:06,413 INFO [IPC Server handler 7 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53333 is added to blk_7946535798164252757_1059 size 21685 2013-05-24 22:01:06,418 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HStore(761): Added hdfs://localhost:54096/user/hortonzy/hbase/test/a88e8c9e73f832adb3635d6996d9e72d/f/cf3fb5911142449890bc91d65bfacf11, entries=703, sequenceid=3791, filesize=21.2 K 2013-05-24 22:01:06,418 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegion(1596): Finished memstore flush of ~115.3 K/118104, currentsize=0/0 for region test,,1369432820315.a88e8c9e73f832adb3635d6996d9e72d. in 405ms, sequenceid=3791, compaction requested=false 2013-05-24 22:01:06,421 INFO [StoreCloserThread-test,,1369432820315.a88e8c9e73f832adb3635d6996d9e72d.-1] regionserver.HStore(663): Closed f 2013-05-24 22:01:06,421 INFO [StoreCloserThread-test,,1369432820315.a88e8c9e73f832adb3635d6996d9e72d.-1] regionserver.HStore(663): Closed norep 2013-05-24 22:01:06,421 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegion(1006): Closed test,,1369432820315.a88e8c9e73f832adb3635d6996d9e72d. 2013-05-24 22:01:06,421 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] handler.CloseRegionHandler(177): Closed region test,,1369432820315.a88e8c9e73f832adb3635d6996d9e72d. 2013-05-24 22:01:06,421 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] handler.CloseRegionHandler(125): Processing close of test,mmm,1369432820320.ab7b9b9773243cff0ac09f3c379e33dc. 2013-05-24 22:01:06,422 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegion(927): Closing test,mmm,1369432820320.ab7b9b9773243cff0ac09f3c379e33dc.: disabling compactions & flushes 2013-05-24 22:01:06,422 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegion(949): Updates disabled for region test,mmm,1369432820320.ab7b9b9773243cff0ac09f3c379e33dc. 2013-05-24 22:01:06,422 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegion(1452): Started memstore flush for test,mmm,1369432820320.ab7b9b9773243cff0ac09f3c379e33dc., current region memstore size 115.3 K 2013-05-24 22:01:06,528 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.DefaultStoreFlusher(88): Flushed, sequenceid=3792, memsize=115.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:54096/user/hortonzy/hbase/test/643c76d94d70be44d35b5a12853bb852/.tmp/3ad1248003294ec19be1af0175ab3c85 2013-05-24 22:01:06,536 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegionFileSystem(338): Committing store file hdfs://localhost:54096/user/hortonzy/hbase/test/643c76d94d70be44d35b5a12853bb852/.tmp/3ad1248003294ec19be1af0175ab3c85 as hdfs://localhost:54096/user/hortonzy/hbase/test/643c76d94d70be44d35b5a12853bb852/f/3ad1248003294ec19be1af0175ab3c85 2013-05-24 22:01:06,578 INFO [IPC Server handler 8 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59993 is added to blk_-9171188685577169369_1060{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:53333|RBW], ReplicaUnderConstruction[127.0.0.1:59993|RBW]]} size 0 2013-05-24 22:01:06,578 INFO [IPC Server handler 4 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53333 is added to blk_-9171188685577169369_1060 size 21685 2013-05-24 22:01:06,648 INFO [IPC Server handler 8 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59993 is added to blk_3316300643951909936_1061{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:53333|RBW], ReplicaUnderConstruction[127.0.0.1:59993|RBW]]} size 0 2013-05-24 22:01:06,649 INFO [IPC Server handler 6 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53333 is added to blk_3316300643951909936_1061 size 21685 2013-05-24 22:01:06,653 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HStore(761): Added hdfs://localhost:54096/user/hortonzy/hbase/test/643c76d94d70be44d35b5a12853bb852/f/3ad1248003294ec19be1af0175ab3c85, entries=703, sequenceid=3792, filesize=21.2 K 2013-05-24 22:01:06,653 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegion(1596): Finished memstore flush of ~115.3 K/118104, currentsize=0/0 for region test,www,1369432820320.643c76d94d70be44d35b5a12853bb852. in 497ms, sequenceid=3792, compaction requested=false 2013-05-24 22:01:06,655 INFO [StoreCloserThread-test,www,1369432820320.643c76d94d70be44d35b5a12853bb852.-1] regionserver.HStore(663): Closed f 2013-05-24 22:01:06,655 INFO [StoreCloserThread-test,www,1369432820320.643c76d94d70be44d35b5a12853bb852.-1] regionserver.HStore(663): Closed norep 2013-05-24 22:01:06,655 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegion(1006): Closed test,www,1369432820320.643c76d94d70be44d35b5a12853bb852. 2013-05-24 22:01:06,655 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] handler.CloseRegionHandler(177): Closed region test,www,1369432820320.643c76d94d70be44d35b5a12853bb852. 2013-05-24 22:01:06,655 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] handler.CloseRegionHandler(125): Processing close of test,eee,1369432820319.6f0a98b71d64131085456015699d44f0. 2013-05-24 22:01:06,656 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegion(927): Closing test,eee,1369432820319.6f0a98b71d64131085456015699d44f0.: disabling compactions & flushes 2013-05-24 22:01:06,656 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegion(949): Updates disabled for region test,eee,1369432820319.6f0a98b71d64131085456015699d44f0. 2013-05-24 22:01:06,656 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegion(1452): Started memstore flush for test,eee,1369432820319.6f0a98b71d64131085456015699d44f0., current region memstore size 115.3 K 2013-05-24 22:01:07,315 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.DefaultStoreFlusher(88): Flushed, sequenceid=3793, memsize=115.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:54096/user/hortonzy/hbase/test/117d9d6b052965f7196396ea21f07a61/.tmp/93c404d976d04710b9a7b5aa21c94ef5 2013-05-24 22:01:07,322 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegionFileSystem(338): Committing store file hdfs://localhost:54096/user/hortonzy/hbase/test/117d9d6b052965f7196396ea21f07a61/.tmp/93c404d976d04710b9a7b5aa21c94ef5 as hdfs://localhost:54096/user/hortonzy/hbase/test/117d9d6b052965f7196396ea21f07a61/f/93c404d976d04710b9a7b5aa21c94ef5 2013-05-24 22:01:07,375 INFO [Master:0;kiyo.gq1.ygridcore.net,49101,1369432806564] master.ServerManager(448): Waiting on regionserver(s) to go down kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:01:07,406 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911] regionserver.HRegionServer(1048): Waiting on 14 regions to close 2013-05-24 22:01:07,492 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.DefaultStoreFlusher(88): Flushed, sequenceid=3794, memsize=115.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:54096/user/hortonzy/hbase/test/ab7b9b9773243cff0ac09f3c379e33dc/.tmp/9a5a5c3addff4f03917f5484bd569438 2013-05-24 22:01:07,500 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegionFileSystem(338): Committing store file hdfs://localhost:54096/user/hortonzy/hbase/test/ab7b9b9773243cff0ac09f3c379e33dc/.tmp/9a5a5c3addff4f03917f5484bd569438 as hdfs://localhost:54096/user/hortonzy/hbase/test/ab7b9b9773243cff0ac09f3c379e33dc/f/9a5a5c3addff4f03917f5484bd569438 2013-05-24 22:01:07,564 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HStore(761): Added hdfs://localhost:54096/user/hortonzy/hbase/test/117d9d6b052965f7196396ea21f07a61/f/93c404d976d04710b9a7b5aa21c94ef5, entries=703, sequenceid=3793, filesize=21.2 K 2013-05-24 22:01:07,564 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(1596): Finished memstore flush of ~115.3 K/118104, currentsize=0/0 for region test,ccc,1369432820319.117d9d6b052965f7196396ea21f07a61. in 1229ms, sequenceid=3793, compaction requested=false 2013-05-24 22:01:07,566 INFO [StoreCloserThread-test,ccc,1369432820319.117d9d6b052965f7196396ea21f07a61.-1] regionserver.HStore(663): Closed f 2013-05-24 22:01:07,567 INFO [StoreCloserThread-test,ccc,1369432820319.117d9d6b052965f7196396ea21f07a61.-1] regionserver.HStore(663): Closed norep 2013-05-24 22:01:07,567 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(1006): Closed test,ccc,1369432820319.117d9d6b052965f7196396ea21f07a61. 2013-05-24 22:01:07,567 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] handler.CloseRegionHandler(177): Closed region test,ccc,1369432820319.117d9d6b052965f7196396ea21f07a61. 2013-05-24 22:01:07,567 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] handler.CloseRegionHandler(125): Processing close of test,hhh,1369432820320.a9a8fa0d1fb6a035656bc264ec313d2b. 2013-05-24 22:01:07,567 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(927): Closing test,hhh,1369432820320.a9a8fa0d1fb6a035656bc264ec313d2b.: disabling compactions & flushes 2013-05-24 22:01:07,567 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(949): Updates disabled for region test,hhh,1369432820320.a9a8fa0d1fb6a035656bc264ec313d2b. 2013-05-24 22:01:07,567 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(1452): Started memstore flush for test,hhh,1369432820320.a9a8fa0d1fb6a035656bc264ec313d2b., current region memstore size 115.3 K 2013-05-24 22:01:07,718 INFO [IPC Server handler 7 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59993 is added to blk_-5306449579699306657_1062{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:53333|RBW], ReplicaUnderConstruction[127.0.0.1:59993|RBW]]} size 0 2013-05-24 22:01:07,719 INFO [IPC Server handler 6 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53333 is added to blk_-5306449579699306657_1062 size 21685 2013-05-24 22:01:07,723 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HStore(761): Added hdfs://localhost:54096/user/hortonzy/hbase/test/ab7b9b9773243cff0ac09f3c379e33dc/f/9a5a5c3addff4f03917f5484bd569438, entries=703, sequenceid=3794, filesize=21.2 K 2013-05-24 22:01:07,724 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegion(1596): Finished memstore flush of ~115.3 K/118104, currentsize=0/0 for region test,mmm,1369432820320.ab7b9b9773243cff0ac09f3c379e33dc. in 1302ms, sequenceid=3794, compaction requested=false 2013-05-24 22:01:07,726 INFO [StoreCloserThread-test,mmm,1369432820320.ab7b9b9773243cff0ac09f3c379e33dc.-1] regionserver.HStore(663): Closed f 2013-05-24 22:01:07,726 INFO [StoreCloserThread-test,mmm,1369432820320.ab7b9b9773243cff0ac09f3c379e33dc.-1] regionserver.HStore(663): Closed norep 2013-05-24 22:01:07,727 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegion(1006): Closed test,mmm,1369432820320.ab7b9b9773243cff0ac09f3c379e33dc. 2013-05-24 22:01:07,727 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] handler.CloseRegionHandler(177): Closed region test,mmm,1369432820320.ab7b9b9773243cff0ac09f3c379e33dc. 2013-05-24 22:01:07,727 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] handler.CloseRegionHandler(125): Processing close of test,nnn,1369432820320.53e3aef4cc8815dd31923659bff5ca21. 2013-05-24 22:01:07,727 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegion(927): Closing test,nnn,1369432820320.53e3aef4cc8815dd31923659bff5ca21.: disabling compactions & flushes 2013-05-24 22:01:07,727 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegion(949): Updates disabled for region test,nnn,1369432820320.53e3aef4cc8815dd31923659bff5ca21. 2013-05-24 22:01:07,728 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegion(1452): Started memstore flush for test,nnn,1369432820320.53e3aef4cc8815dd31923659bff5ca21., current region memstore size 115.3 K 2013-05-24 22:01:08,061 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.DefaultStoreFlusher(88): Flushed, sequenceid=3795, memsize=115.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:54096/user/hortonzy/hbase/test/6f0a98b71d64131085456015699d44f0/.tmp/f0fdd99a947e467bbb40ba885d9f4183 2013-05-24 22:01:08,068 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegionFileSystem(338): Committing store file hdfs://localhost:54096/user/hortonzy/hbase/test/6f0a98b71d64131085456015699d44f0/.tmp/f0fdd99a947e467bbb40ba885d9f4183 as hdfs://localhost:54096/user/hortonzy/hbase/test/6f0a98b71d64131085456015699d44f0/f/f0fdd99a947e467bbb40ba885d9f4183 2013-05-24 22:01:08,121 INFO [IPC Server handler 7 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53333 is added to blk_-2977549211690159164_1063{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:59993|RBW], ReplicaUnderConstruction[127.0.0.1:53333|RBW]]} size 0 2013-05-24 22:01:08,122 INFO [IPC Server handler 1 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59993 is added to blk_-2977549211690159164_1063 size 21685 2013-05-24 22:01:08,140 INFO [IPC Server handler 7 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59993 is added to blk_4843355520008312098_1064{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:53333|RBW], ReplicaUnderConstruction[127.0.0.1:59993|RBW]]} size 0 2013-05-24 22:01:08,141 INFO [IPC Server handler 0 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53333 is added to blk_4843355520008312098_1064 size 21685 2013-05-24 22:01:08,144 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HStore(761): Added hdfs://localhost:54096/user/hortonzy/hbase/test/6f0a98b71d64131085456015699d44f0/f/f0fdd99a947e467bbb40ba885d9f4183, entries=703, sequenceid=3795, filesize=21.2 K 2013-05-24 22:01:08,145 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegion(1596): Finished memstore flush of ~115.3 K/118104, currentsize=0/0 for region test,eee,1369432820319.6f0a98b71d64131085456015699d44f0. in 1489ms, sequenceid=3795, compaction requested=false 2013-05-24 22:01:08,147 INFO [StoreCloserThread-test,eee,1369432820319.6f0a98b71d64131085456015699d44f0.-1] regionserver.HStore(663): Closed f 2013-05-24 22:01:08,148 INFO [StoreCloserThread-test,eee,1369432820319.6f0a98b71d64131085456015699d44f0.-1] regionserver.HStore(663): Closed norep 2013-05-24 22:01:08,148 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegion(1006): Closed test,eee,1369432820319.6f0a98b71d64131085456015699d44f0. 2013-05-24 22:01:08,148 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] handler.CloseRegionHandler(177): Closed region test,eee,1369432820319.6f0a98b71d64131085456015699d44f0. 2013-05-24 22:01:08,148 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] handler.CloseRegionHandler(125): Processing close of test,ddd,1369432820319.925080806d078451834e3417849dd4a9. 2013-05-24 22:01:08,148 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegion(927): Closing test,ddd,1369432820319.925080806d078451834e3417849dd4a9.: disabling compactions & flushes 2013-05-24 22:01:08,148 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegion(949): Updates disabled for region test,ddd,1369432820319.925080806d078451834e3417849dd4a9. 2013-05-24 22:01:08,148 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegion(1452): Started memstore flush for test,ddd,1369432820319.925080806d078451834e3417849dd4a9., current region memstore size 115.3 K 2013-05-24 22:01:08,197 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.DefaultStoreFlusher(88): Flushed, sequenceid=3796, memsize=115.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:54096/user/hortonzy/hbase/test/a9a8fa0d1fb6a035656bc264ec313d2b/.tmp/51931188d94640fd899d1ca586e0eee3 2013-05-24 22:01:08,205 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegionFileSystem(338): Committing store file hdfs://localhost:54096/user/hortonzy/hbase/test/a9a8fa0d1fb6a035656bc264ec313d2b/.tmp/51931188d94640fd899d1ca586e0eee3 as hdfs://localhost:54096/user/hortonzy/hbase/test/a9a8fa0d1fb6a035656bc264ec313d2b/f/51931188d94640fd899d1ca586e0eee3 2013-05-24 22:01:08,319 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.DefaultStoreFlusher(88): Flushed, sequenceid=3797, memsize=115.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:54096/user/hortonzy/hbase/test/53e3aef4cc8815dd31923659bff5ca21/.tmp/b9d7f40f17224d77918734f036ed9e15 2013-05-24 22:01:08,326 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegionFileSystem(338): Committing store file hdfs://localhost:54096/user/hortonzy/hbase/test/53e3aef4cc8815dd31923659bff5ca21/.tmp/b9d7f40f17224d77918734f036ed9e15 as hdfs://localhost:54096/user/hortonzy/hbase/test/53e3aef4cc8815dd31923659bff5ca21/f/b9d7f40f17224d77918734f036ed9e15 2013-05-24 22:01:08,342 INFO [kiyo.gq1.ygridcore.net,39146,1369432814418Replication Statistics #0] regionserver.Replication$ReplicationStatisticsThread(291): Sink: age in ms of last applied edit: 33247, total replicated edits: 2115 2013-05-24 22:01:08,377 INFO [Master:0;kiyo.gq1.ygridcore.net,49101,1369432806564] master.ServerManager(448): Waiting on regionserver(s) to go down kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:01:08,399 INFO [kiyo.gq1.ygridcore.net,51523,1369432814392Replication Statistics #0] regionserver.Replication$ReplicationStatisticsThread(291): Sink: age in ms of last applied edit: 26571, total replicated edits: 3593 2013-05-24 22:01:08,408 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911] regionserver.HRegionServer(1048): Waiting on 11 regions to close 2013-05-24 22:01:08,657 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HStore(761): Added hdfs://localhost:54096/user/hortonzy/hbase/test/a9a8fa0d1fb6a035656bc264ec313d2b/f/51931188d94640fd899d1ca586e0eee3, entries=703, sequenceid=3796, filesize=21.2 K 2013-05-24 22:01:08,657 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(1596): Finished memstore flush of ~115.3 K/118104, currentsize=0/0 for region test,hhh,1369432820320.a9a8fa0d1fb6a035656bc264ec313d2b. in 1090ms, sequenceid=3796, compaction requested=false 2013-05-24 22:01:08,659 INFO [StoreCloserThread-test,hhh,1369432820320.a9a8fa0d1fb6a035656bc264ec313d2b.-1] regionserver.HStore(663): Closed f 2013-05-24 22:01:08,660 INFO [StoreCloserThread-test,hhh,1369432820320.a9a8fa0d1fb6a035656bc264ec313d2b.-1] regionserver.HStore(663): Closed norep 2013-05-24 22:01:08,660 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(1006): Closed test,hhh,1369432820320.a9a8fa0d1fb6a035656bc264ec313d2b. 2013-05-24 22:01:08,660 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] handler.CloseRegionHandler(177): Closed region test,hhh,1369432820320.a9a8fa0d1fb6a035656bc264ec313d2b. 2013-05-24 22:01:08,660 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] handler.CloseRegionHandler(125): Processing close of test,qqq,1369432820320.ec0b9e20bb8d4dc8fee9b3e80c5cae32. 2013-05-24 22:01:08,660 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(927): Closing test,qqq,1369432820320.ec0b9e20bb8d4dc8fee9b3e80c5cae32.: disabling compactions & flushes 2013-05-24 22:01:08,660 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(949): Updates disabled for region test,qqq,1369432820320.ec0b9e20bb8d4dc8fee9b3e80c5cae32. 2013-05-24 22:01:08,660 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(1452): Started memstore flush for test,qqq,1369432820320.ec0b9e20bb8d4dc8fee9b3e80c5cae32., current region memstore size 115.3 K 2013-05-24 22:01:08,686 INFO [Master:0;kiyo.gq1.ygridcore.net,49101,1369432806564.oldLogCleaner] hbase.Chore(93): Master:0;kiyo.gq1.ygridcore.net,49101,1369432806564.oldLogCleaner exiting 2013-05-24 22:01:08,687 INFO [Master:0;kiyo.gq1.ygridcore.net,49101,1369432806564.oldLogCleaner] master.ReplicationLogCleaner(154): Stopping replicationLogCleaner-0x13ed88e2cdf000a 2013-05-24 22:01:08,692 INFO [Master:0;kiyo.gq1.ygridcore.net,49101,1369432806564.archivedHFileCleaner] hbase.Chore(93): Master:0;kiyo.gq1.ygridcore.net,49101,1369432806564.archivedHFileCleaner exiting 2013-05-24 22:01:08,727 INFO [Master:0;kiyo.gq1.ygridcore.net,49101,1369432806564.oldLogCleaner] master.ReplicationLogCleaner(158): Stopping org.apache.hadoop.hbase.replication.ReplicationStateImpl@63ed20be 2013-05-24 22:01:08,727 ERROR [Master:0;kiyo.gq1.ygridcore.net,49101,1369432806564.oldLogCleaner] client.HConnectionManager(362): Connection not found in the list, can't delete it (connection key=HConnectionKey{properties={hbase.rpc.timeout=60000, hbase.zookeeper.property.clientPort=51411, hbase.client.pause=1000, zookeeper.znode.parent=/1, hbase.client.retries.number=100, hbase.zookeeper.quorum=localhost}, username='hortonzy'}). May be the key was modified? 2013-05-24 22:01:08,730 INFO [IPC Server handler 8 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53333 is added to blk_-4109866646414874825_1065{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:59993|RBW], ReplicaUnderConstruction[127.0.0.1:53333|RBW]]} size 0 2013-05-24 22:01:08,730 INFO [IPC Server handler 0 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59993 is added to blk_-4109866646414874825_1065 size 21685 2013-05-24 22:01:08,735 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HStore(761): Added hdfs://localhost:54096/user/hortonzy/hbase/test/53e3aef4cc8815dd31923659bff5ca21/f/b9d7f40f17224d77918734f036ed9e15, entries=703, sequenceid=3797, filesize=21.2 K 2013-05-24 22:01:08,737 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegion(1596): Finished memstore flush of ~115.3 K/118104, currentsize=0/0 for region test,nnn,1369432820320.53e3aef4cc8815dd31923659bff5ca21. in 1008ms, sequenceid=3797, compaction requested=false 2013-05-24 22:01:08,742 INFO [StoreCloserThread-test,nnn,1369432820320.53e3aef4cc8815dd31923659bff5ca21.-1] regionserver.HStore(663): Closed f 2013-05-24 22:01:08,742 INFO [StoreCloserThread-test,nnn,1369432820320.53e3aef4cc8815dd31923659bff5ca21.-1] regionserver.HStore(663): Closed norep 2013-05-24 22:01:08,743 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegion(1006): Closed test,nnn,1369432820320.53e3aef4cc8815dd31923659bff5ca21. 2013-05-24 22:01:08,743 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] handler.CloseRegionHandler(177): Closed region test,nnn,1369432820320.53e3aef4cc8815dd31923659bff5ca21. 2013-05-24 22:01:08,743 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] handler.CloseRegionHandler(125): Processing close of test,ppp,1369432820320.da89687648e34409286fd02380093aed. 2013-05-24 22:01:08,743 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegion(927): Closing test,ppp,1369432820320.da89687648e34409286fd02380093aed.: disabling compactions & flushes 2013-05-24 22:01:08,743 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegion(949): Updates disabled for region test,ppp,1369432820320.da89687648e34409286fd02380093aed. 2013-05-24 22:01:08,744 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegion(1452): Started memstore flush for test,ppp,1369432820320.da89687648e34409286fd02380093aed., current region memstore size 115.3 K 2013-05-24 22:01:08,781 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.DefaultStoreFlusher(88): Flushed, sequenceid=3798, memsize=115.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:54096/user/hortonzy/hbase/test/925080806d078451834e3417849dd4a9/.tmp/210d9f294c6c45feaded3c8711562287 2013-05-24 22:01:08,789 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegionFileSystem(338): Committing store file hdfs://localhost:54096/user/hortonzy/hbase/test/925080806d078451834e3417849dd4a9/.tmp/210d9f294c6c45feaded3c8711562287 as hdfs://localhost:54096/user/hortonzy/hbase/test/925080806d078451834e3417849dd4a9/f/210d9f294c6c45feaded3c8711562287 2013-05-24 22:01:09,028 INFO [IPC Server handler 0 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53333 is added to blk_5498546615314697997_1066{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:59993|RBW], ReplicaUnderConstruction[127.0.0.1:53333|RBW]]} size 0 2013-05-24 22:01:09,029 INFO [IPC Server handler 7 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59993 is added to blk_5498546615314697997_1066 size 21685 2013-05-24 22:01:09,197 INFO [IPC Server handler 0 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59993 is added to blk_8983265313066871417_1067{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:53333|RBW], ReplicaUnderConstruction[127.0.0.1:59993|RBW]]} size 0 2013-05-24 22:01:09,198 INFO [IPC Server handler 5 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53333 is added to blk_8983265313066871417_1067 size 21685 2013-05-24 22:01:09,202 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HStore(761): Added hdfs://localhost:54096/user/hortonzy/hbase/test/925080806d078451834e3417849dd4a9/f/210d9f294c6c45feaded3c8711562287, entries=703, sequenceid=3798, filesize=21.2 K 2013-05-24 22:01:09,202 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegion(1596): Finished memstore flush of ~115.3 K/118104, currentsize=0/0 for region test,ddd,1369432820319.925080806d078451834e3417849dd4a9. in 1054ms, sequenceid=3798, compaction requested=false 2013-05-24 22:01:09,205 INFO [StoreCloserThread-test,ddd,1369432820319.925080806d078451834e3417849dd4a9.-1] regionserver.HStore(663): Closed f 2013-05-24 22:01:09,205 INFO [StoreCloserThread-test,ddd,1369432820319.925080806d078451834e3417849dd4a9.-1] regionserver.HStore(663): Closed norep 2013-05-24 22:01:09,205 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegion(1006): Closed test,ddd,1369432820319.925080806d078451834e3417849dd4a9. 2013-05-24 22:01:09,205 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] handler.CloseRegionHandler(177): Closed region test,ddd,1369432820319.925080806d078451834e3417849dd4a9. 2013-05-24 22:01:09,205 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] handler.CloseRegionHandler(125): Processing close of test,lll,1369432820320.307a0833b0558971bccf01de5c996ebc. 2013-05-24 22:01:09,205 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegion(927): Closing test,lll,1369432820320.307a0833b0558971bccf01de5c996ebc.: disabling compactions & flushes 2013-05-24 22:01:09,206 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegion(949): Updates disabled for region test,lll,1369432820320.307a0833b0558971bccf01de5c996ebc. 2013-05-24 22:01:09,206 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegion(1452): Started memstore flush for test,lll,1369432820320.307a0833b0558971bccf01de5c996ebc., current region memstore size 115.3 K 2013-05-24 22:01:09,222 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.DefaultStoreFlusher(88): Flushed, sequenceid=3799, memsize=115.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:54096/user/hortonzy/hbase/test/ec0b9e20bb8d4dc8fee9b3e80c5cae32/.tmp/df34e85b547b400f90c4ef7e3acb379e 2013-05-24 22:01:09,229 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegionFileSystem(338): Committing store file hdfs://localhost:54096/user/hortonzy/hbase/test/ec0b9e20bb8d4dc8fee9b3e80c5cae32/.tmp/df34e85b547b400f90c4ef7e3acb379e as hdfs://localhost:54096/user/hortonzy/hbase/test/ec0b9e20bb8d4dc8fee9b3e80c5cae32/f/df34e85b547b400f90c4ef7e3acb379e 2013-05-24 22:01:09,373 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.DefaultStoreFlusher(88): Flushed, sequenceid=3800, memsize=115.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:54096/user/hortonzy/hbase/test/da89687648e34409286fd02380093aed/.tmp/da65e2c5d1944d2588a70f4b3b38ee77 2013-05-24 22:01:09,379 INFO [Master:0;kiyo.gq1.ygridcore.net,49101,1369432806564] master.ServerManager(448): Waiting on regionserver(s) to go down kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:01:09,381 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegionFileSystem(338): Committing store file hdfs://localhost:54096/user/hortonzy/hbase/test/da89687648e34409286fd02380093aed/.tmp/da65e2c5d1944d2588a70f4b3b38ee77 as hdfs://localhost:54096/user/hortonzy/hbase/test/da89687648e34409286fd02380093aed/f/da65e2c5d1944d2588a70f4b3b38ee77 2013-05-24 22:01:09,394 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HStore(761): Added hdfs://localhost:54096/user/hortonzy/hbase/test/ec0b9e20bb8d4dc8fee9b3e80c5cae32/f/df34e85b547b400f90c4ef7e3acb379e, entries=703, sequenceid=3799, filesize=21.2 K 2013-05-24 22:01:09,395 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(1596): Finished memstore flush of ~115.3 K/118104, currentsize=0/0 for region test,qqq,1369432820320.ec0b9e20bb8d4dc8fee9b3e80c5cae32. in 734ms, sequenceid=3799, compaction requested=false 2013-05-24 22:01:09,396 INFO [StoreCloserThread-test,qqq,1369432820320.ec0b9e20bb8d4dc8fee9b3e80c5cae32.-1] regionserver.HStore(663): Closed f 2013-05-24 22:01:09,396 INFO [StoreCloserThread-test,qqq,1369432820320.ec0b9e20bb8d4dc8fee9b3e80c5cae32.-1] regionserver.HStore(663): Closed norep 2013-05-24 22:01:09,397 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(1006): Closed test,qqq,1369432820320.ec0b9e20bb8d4dc8fee9b3e80c5cae32. 2013-05-24 22:01:09,397 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] handler.CloseRegionHandler(177): Closed region test,qqq,1369432820320.ec0b9e20bb8d4dc8fee9b3e80c5cae32. 2013-05-24 22:01:09,397 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] handler.CloseRegionHandler(125): Processing close of test,bbb,1369432820319.6a4e618fc444a41a507e36bb35c860a6. 2013-05-24 22:01:09,397 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(927): Closing test,bbb,1369432820319.6a4e618fc444a41a507e36bb35c860a6.: disabling compactions & flushes 2013-05-24 22:01:09,397 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(949): Updates disabled for region test,bbb,1369432820319.6a4e618fc444a41a507e36bb35c860a6. 2013-05-24 22:01:09,397 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(1452): Started memstore flush for test,bbb,1369432820319.6a4e618fc444a41a507e36bb35c860a6., current region memstore size 115.3 K 2013-05-24 22:01:09,409 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911] regionserver.HRegionServer(1048): Waiting on 7 regions to close 2013-05-24 22:01:09,409 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911] regionserver.HRegionServer(1052): {da89687648e34409286fd02380093aed=test,ppp,1369432820320.da89687648e34409286fd02380093aed., 307a0833b0558971bccf01de5c996ebc=test,lll,1369432820320.307a0833b0558971bccf01de5c996ebc., 6a4e618fc444a41a507e36bb35c860a6=test,bbb,1369432820319.6a4e618fc444a41a507e36bb35c860a6., eb38c552d2a930aa1ce862202da80908=test,jjj,1369432820320.eb38c552d2a930aa1ce862202da80908., b03c7b8b6013fa49f346edeb3a7cad87=test,xxx,1369432820320.b03c7b8b6013fa49f346edeb3a7cad87., 84d2bf1edb7e9df4777e80207fc739d2=test,vvv,1369432820320.84d2bf1edb7e9df4777e80207fc739d2., f53d942877419dd243e3fb7e46c2cf97=test,ttt,1369432820320.f53d942877419dd243e3fb7e46c2cf97.} 2013-05-24 22:01:09,432 INFO [IPC Server handler 3 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53333 is added to blk_-7197039210025190107_1068{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:59993|RBW], ReplicaUnderConstruction[127.0.0.1:53333|RBW]]} size 0 2013-05-24 22:01:09,433 INFO [IPC Server handler 5 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59993 is added to blk_-7197039210025190107_1068 size 21685 2013-05-24 22:01:09,437 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HStore(761): Added hdfs://localhost:54096/user/hortonzy/hbase/test/da89687648e34409286fd02380093aed/f/da65e2c5d1944d2588a70f4b3b38ee77, entries=703, sequenceid=3800, filesize=21.2 K 2013-05-24 22:01:09,438 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegion(1596): Finished memstore flush of ~115.3 K/118104, currentsize=0/0 for region test,ppp,1369432820320.da89687648e34409286fd02380093aed. in 695ms, sequenceid=3800, compaction requested=false 2013-05-24 22:01:09,440 INFO [StoreCloserThread-test,ppp,1369432820320.da89687648e34409286fd02380093aed.-1] regionserver.HStore(663): Closed f 2013-05-24 22:01:09,440 INFO [StoreCloserThread-test,ppp,1369432820320.da89687648e34409286fd02380093aed.-1] regionserver.HStore(663): Closed norep 2013-05-24 22:01:09,440 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegion(1006): Closed test,ppp,1369432820320.da89687648e34409286fd02380093aed. 2013-05-24 22:01:09,441 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] handler.CloseRegionHandler(177): Closed region test,ppp,1369432820320.da89687648e34409286fd02380093aed. 2013-05-24 22:01:09,441 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] handler.CloseRegionHandler(125): Processing close of test,jjj,1369432820320.eb38c552d2a930aa1ce862202da80908. 2013-05-24 22:01:09,441 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegion(927): Closing test,jjj,1369432820320.eb38c552d2a930aa1ce862202da80908.: disabling compactions & flushes 2013-05-24 22:01:09,441 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegion(949): Updates disabled for region test,jjj,1369432820320.eb38c552d2a930aa1ce862202da80908. 2013-05-24 22:01:09,441 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegion(1452): Started memstore flush for test,jjj,1369432820320.eb38c552d2a930aa1ce862202da80908., current region memstore size 115.3 K 2013-05-24 22:01:09,472 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.DefaultStoreFlusher(88): Flushed, sequenceid=3801, memsize=115.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:54096/user/hortonzy/hbase/test/307a0833b0558971bccf01de5c996ebc/.tmp/afc8f1fee0834413880338f5e86ac992 2013-05-24 22:01:09,479 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegionFileSystem(338): Committing store file hdfs://localhost:54096/user/hortonzy/hbase/test/307a0833b0558971bccf01de5c996ebc/.tmp/afc8f1fee0834413880338f5e86ac992 as hdfs://localhost:54096/user/hortonzy/hbase/test/307a0833b0558971bccf01de5c996ebc/f/afc8f1fee0834413880338f5e86ac992 2013-05-24 22:01:09,495 INFO [IPC Server handler 3 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59993 is added to blk_-7506501999393291381_1069{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:53333|RBW], ReplicaUnderConstruction[127.0.0.1:59993|RBW]]} size 0 2013-05-24 22:01:09,496 INFO [IPC Server handler 5 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53333 is added to blk_-7506501999393291381_1069 size 21685 2013-05-24 22:01:09,551 INFO [IPC Server handler 3 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59993 is added to blk_8427357440661633896_1070{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:53333|RBW], ReplicaUnderConstruction[127.0.0.1:59993|RBW]]} size 0 2013-05-24 22:01:09,552 INFO [IPC Server handler 6 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53333 is added to blk_8427357440661633896_1070 size 21685 2013-05-24 22:01:09,556 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HStore(761): Added hdfs://localhost:54096/user/hortonzy/hbase/test/307a0833b0558971bccf01de5c996ebc/f/afc8f1fee0834413880338f5e86ac992, entries=703, sequenceid=3801, filesize=21.2 K 2013-05-24 22:01:09,556 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegion(1596): Finished memstore flush of ~115.3 K/118104, currentsize=0/0 for region test,lll,1369432820320.307a0833b0558971bccf01de5c996ebc. in 350ms, sequenceid=3801, compaction requested=false 2013-05-24 22:01:09,558 INFO [StoreCloserThread-test,lll,1369432820320.307a0833b0558971bccf01de5c996ebc.-1] regionserver.HStore(663): Closed f 2013-05-24 22:01:09,558 INFO [StoreCloserThread-test,lll,1369432820320.307a0833b0558971bccf01de5c996ebc.-1] regionserver.HStore(663): Closed norep 2013-05-24 22:01:09,558 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegion(1006): Closed test,lll,1369432820320.307a0833b0558971bccf01de5c996ebc. 2013-05-24 22:01:09,558 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] handler.CloseRegionHandler(177): Closed region test,lll,1369432820320.307a0833b0558971bccf01de5c996ebc. 2013-05-24 22:01:09,558 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] handler.CloseRegionHandler(125): Processing close of test,xxx,1369432820320.b03c7b8b6013fa49f346edeb3a7cad87. 2013-05-24 22:01:09,559 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegion(927): Closing test,xxx,1369432820320.b03c7b8b6013fa49f346edeb3a7cad87.: disabling compactions & flushes 2013-05-24 22:01:09,559 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegion(949): Updates disabled for region test,xxx,1369432820320.b03c7b8b6013fa49f346edeb3a7cad87. 2013-05-24 22:01:09,559 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegion(1452): Started memstore flush for test,xxx,1369432820320.b03c7b8b6013fa49f346edeb3a7cad87., current region memstore size 115.3 K 2013-05-24 22:01:09,563 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.DefaultStoreFlusher(88): Flushed, sequenceid=3802, memsize=115.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:54096/user/hortonzy/hbase/test/6a4e618fc444a41a507e36bb35c860a6/.tmp/e10e1679435d469d82c2aa111a5d993d 2013-05-24 22:01:09,570 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegionFileSystem(338): Committing store file hdfs://localhost:54096/user/hortonzy/hbase/test/6a4e618fc444a41a507e36bb35c860a6/.tmp/e10e1679435d469d82c2aa111a5d993d as hdfs://localhost:54096/user/hortonzy/hbase/test/6a4e618fc444a41a507e36bb35c860a6/f/e10e1679435d469d82c2aa111a5d993d 2013-05-24 22:01:09,588 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.DefaultStoreFlusher(88): Flushed, sequenceid=3803, memsize=115.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:54096/user/hortonzy/hbase/test/eb38c552d2a930aa1ce862202da80908/.tmp/7d94c64b80f34edf8b1674fe574a1522 2013-05-24 22:01:09,597 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegionFileSystem(338): Committing store file hdfs://localhost:54096/user/hortonzy/hbase/test/eb38c552d2a930aa1ce862202da80908/.tmp/7d94c64b80f34edf8b1674fe574a1522 as hdfs://localhost:54096/user/hortonzy/hbase/test/eb38c552d2a930aa1ce862202da80908/f/7d94c64b80f34edf8b1674fe574a1522 2013-05-24 22:01:09,620 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HStore(761): Added hdfs://localhost:54096/user/hortonzy/hbase/test/6a4e618fc444a41a507e36bb35c860a6/f/e10e1679435d469d82c2aa111a5d993d, entries=703, sequenceid=3802, filesize=21.2 K 2013-05-24 22:01:09,620 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(1596): Finished memstore flush of ~115.3 K/118104, currentsize=0/0 for region test,bbb,1369432820319.6a4e618fc444a41a507e36bb35c860a6. in 223ms, sequenceid=3802, compaction requested=false 2013-05-24 22:01:09,622 INFO [StoreCloserThread-test,bbb,1369432820319.6a4e618fc444a41a507e36bb35c860a6.-1] regionserver.HStore(663): Closed f 2013-05-24 22:01:09,622 INFO [StoreCloserThread-test,bbb,1369432820319.6a4e618fc444a41a507e36bb35c860a6.-1] regionserver.HStore(663): Closed norep 2013-05-24 22:01:09,622 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(1006): Closed test,bbb,1369432820319.6a4e618fc444a41a507e36bb35c860a6. 2013-05-24 22:01:09,622 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] handler.CloseRegionHandler(177): Closed region test,bbb,1369432820319.6a4e618fc444a41a507e36bb35c860a6. 2013-05-24 22:01:09,622 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] handler.CloseRegionHandler(125): Processing close of test,vvv,1369432820320.84d2bf1edb7e9df4777e80207fc739d2. 2013-05-24 22:01:09,623 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(927): Closing test,vvv,1369432820320.84d2bf1edb7e9df4777e80207fc739d2.: disabling compactions & flushes 2013-05-24 22:01:09,623 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(949): Updates disabled for region test,vvv,1369432820320.84d2bf1edb7e9df4777e80207fc739d2. 2013-05-24 22:01:09,623 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(1452): Started memstore flush for test,vvv,1369432820320.84d2bf1edb7e9df4777e80207fc739d2., current region memstore size 115.3 K 2013-05-24 22:01:09,645 INFO [IPC Server handler 0 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53333 is added to blk_7612205052974946670_1071{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:59993|RBW], ReplicaUnderConstruction[127.0.0.1:53333|RBW]]} size 0 2013-05-24 22:01:09,646 INFO [IPC Server handler 1 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59993 is added to blk_7612205052974946670_1071 size 21685 2013-05-24 22:01:09,650 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HStore(761): Added hdfs://localhost:54096/user/hortonzy/hbase/test/eb38c552d2a930aa1ce862202da80908/f/7d94c64b80f34edf8b1674fe574a1522, entries=703, sequenceid=3803, filesize=21.2 K 2013-05-24 22:01:09,650 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegion(1596): Finished memstore flush of ~115.3 K/118104, currentsize=0/0 for region test,jjj,1369432820320.eb38c552d2a930aa1ce862202da80908. in 209ms, sequenceid=3803, compaction requested=false 2013-05-24 22:01:09,651 INFO [StoreCloserThread-test,jjj,1369432820320.eb38c552d2a930aa1ce862202da80908.-1] regionserver.HStore(663): Closed f 2013-05-24 22:01:09,652 INFO [StoreCloserThread-test,jjj,1369432820320.eb38c552d2a930aa1ce862202da80908.-1] regionserver.HStore(663): Closed norep 2013-05-24 22:01:09,652 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegion(1006): Closed test,jjj,1369432820320.eb38c552d2a930aa1ce862202da80908. 2013-05-24 22:01:09,652 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] handler.CloseRegionHandler(177): Closed region test,jjj,1369432820320.eb38c552d2a930aa1ce862202da80908. 2013-05-24 22:01:09,652 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] handler.CloseRegionHandler(125): Processing close of test,ttt,1369432820320.f53d942877419dd243e3fb7e46c2cf97. 2013-05-24 22:01:09,652 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegion(927): Closing test,ttt,1369432820320.f53d942877419dd243e3fb7e46c2cf97.: disabling compactions & flushes 2013-05-24 22:01:09,652 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegion(949): Updates disabled for region test,ttt,1369432820320.f53d942877419dd243e3fb7e46c2cf97. 2013-05-24 22:01:09,653 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegion(1452): Started memstore flush for test,ttt,1369432820320.f53d942877419dd243e3fb7e46c2cf97., current region memstore size 115.3 K 2013-05-24 22:01:09,680 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.DefaultStoreFlusher(88): Flushed, sequenceid=3804, memsize=115.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:54096/user/hortonzy/hbase/test/b03c7b8b6013fa49f346edeb3a7cad87/.tmp/22b77dc3ad9041049aabfe8611865b84 2013-05-24 22:01:09,687 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegionFileSystem(338): Committing store file hdfs://localhost:54096/user/hortonzy/hbase/test/b03c7b8b6013fa49f346edeb3a7cad87/.tmp/22b77dc3ad9041049aabfe8611865b84 as hdfs://localhost:54096/user/hortonzy/hbase/test/b03c7b8b6013fa49f346edeb3a7cad87/f/22b77dc3ad9041049aabfe8611865b84 2013-05-24 22:01:09,754 INFO [IPC Server handler 0 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59993 is added to blk_-2336005231836988085_1072{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:53333|RBW], ReplicaUnderConstruction[127.0.0.1:59993|RBW]]} size 0 2013-05-24 22:01:09,754 INFO [IPC Server handler 9 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53333 is added to blk_-2336005231836988085_1072 size 21685 2013-05-24 22:01:09,787 INFO [IPC Server handler 0 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59993 is added to blk_-6610240801031680425_1073{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:53333|RBW], ReplicaUnderConstruction[127.0.0.1:59993|RBW]]} size 0 2013-05-24 22:01:09,788 INFO [IPC Server handler 7 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53333 is added to blk_-6610240801031680425_1073 size 21685 2013-05-24 22:01:09,791 INFO [kiyo.gq1.ygridcore.net,42690,1369432806911Replication Statistics #0] regionserver.Replication$ReplicationStatisticsThread(291): Normal source for cluster 2: Total replicated edits: 2667, currently replicating from: hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432840428 at position: N/A 2013-05-24 22:01:09,791 INFO [kiyo.gq1.ygridcore.net,49701,1369432806997Replication Statistics #0] regionserver.Replication$ReplicationStatisticsThread(291): Normal source for cluster 2: Total replicated edits: 1629, currently replicating from: hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,49701,1369432806997/kiyo.gq1.ygridcore.net%2C49701%2C1369432806997.1369432832342 at position: N/A 2013-05-24 22:01:09,791 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HStore(761): Added hdfs://localhost:54096/user/hortonzy/hbase/test/b03c7b8b6013fa49f346edeb3a7cad87/f/22b77dc3ad9041049aabfe8611865b84, entries=703, sequenceid=3804, filesize=21.2 K 2013-05-24 22:01:09,792 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegion(1596): Finished memstore flush of ~115.3 K/118104, currentsize=0/0 for region test,xxx,1369432820320.b03c7b8b6013fa49f346edeb3a7cad87. in 233ms, sequenceid=3804, compaction requested=false 2013-05-24 22:01:09,794 INFO [StoreCloserThread-test,xxx,1369432820320.b03c7b8b6013fa49f346edeb3a7cad87.-1] regionserver.HStore(663): Closed f 2013-05-24 22:01:09,794 INFO [StoreCloserThread-test,xxx,1369432820320.b03c7b8b6013fa49f346edeb3a7cad87.-1] regionserver.HStore(663): Closed norep 2013-05-24 22:01:09,794 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] regionserver.HRegion(1006): Closed test,xxx,1369432820320.b03c7b8b6013fa49f346edeb3a7cad87. 2013-05-24 22:01:09,794 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-1] handler.CloseRegionHandler(177): Closed region test,xxx,1369432820320.b03c7b8b6013fa49f346edeb3a7cad87. 2013-05-24 22:01:09,829 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.DefaultStoreFlusher(88): Flushed, sequenceid=3805, memsize=115.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:54096/user/hortonzy/hbase/test/84d2bf1edb7e9df4777e80207fc739d2/.tmp/7073c638601743f7afbf7b4b5854e1f2 2013-05-24 22:01:09,836 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegionFileSystem(338): Committing store file hdfs://localhost:54096/user/hortonzy/hbase/test/84d2bf1edb7e9df4777e80207fc739d2/.tmp/7073c638601743f7afbf7b4b5854e1f2 as hdfs://localhost:54096/user/hortonzy/hbase/test/84d2bf1edb7e9df4777e80207fc739d2/f/7073c638601743f7afbf7b4b5854e1f2 2013-05-24 22:01:09,862 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.DefaultStoreFlusher(88): Flushed, sequenceid=3806, memsize=115.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:54096/user/hortonzy/hbase/test/f53d942877419dd243e3fb7e46c2cf97/.tmp/1d7718da780547299209a98bc5452665 2013-05-24 22:01:09,870 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegionFileSystem(338): Committing store file hdfs://localhost:54096/user/hortonzy/hbase/test/f53d942877419dd243e3fb7e46c2cf97/.tmp/1d7718da780547299209a98bc5452665 as hdfs://localhost:54096/user/hortonzy/hbase/test/f53d942877419dd243e3fb7e46c2cf97/f/1d7718da780547299209a98bc5452665 2013-05-24 22:01:09,908 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HStore(761): Added hdfs://localhost:54096/user/hortonzy/hbase/test/84d2bf1edb7e9df4777e80207fc739d2/f/7073c638601743f7afbf7b4b5854e1f2, entries=703, sequenceid=3805, filesize=21.2 K 2013-05-24 22:01:09,909 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(1596): Finished memstore flush of ~115.3 K/118104, currentsize=0/0 for region test,vvv,1369432820320.84d2bf1edb7e9df4777e80207fc739d2. in 285ms, sequenceid=3805, compaction requested=false 2013-05-24 22:01:09,910 INFO [StoreCloserThread-test,vvv,1369432820320.84d2bf1edb7e9df4777e80207fc739d2.-1] regionserver.HStore(663): Closed f 2013-05-24 22:01:09,911 INFO [StoreCloserThread-test,vvv,1369432820320.84d2bf1edb7e9df4777e80207fc739d2.-1] regionserver.HStore(663): Closed norep 2013-05-24 22:01:09,911 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] regionserver.HRegion(1006): Closed test,vvv,1369432820320.84d2bf1edb7e9df4777e80207fc739d2. 2013-05-24 22:01:09,911 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-0] handler.CloseRegionHandler(177): Closed region test,vvv,1369432820320.84d2bf1edb7e9df4777e80207fc739d2. 2013-05-24 22:01:09,952 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HStore(761): Added hdfs://localhost:54096/user/hortonzy/hbase/test/f53d942877419dd243e3fb7e46c2cf97/f/1d7718da780547299209a98bc5452665, entries=703, sequenceid=3806, filesize=21.2 K 2013-05-24 22:01:09,952 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegion(1596): Finished memstore flush of ~115.3 K/118104, currentsize=0/0 for region test,ttt,1369432820320.f53d942877419dd243e3fb7e46c2cf97. in 300ms, sequenceid=3806, compaction requested=false 2013-05-24 22:01:09,954 INFO [StoreCloserThread-test,ttt,1369432820320.f53d942877419dd243e3fb7e46c2cf97.-1] regionserver.HStore(663): Closed f 2013-05-24 22:01:09,954 INFO [StoreCloserThread-test,ttt,1369432820320.f53d942877419dd243e3fb7e46c2cf97.-1] regionserver.HStore(663): Closed norep 2013-05-24 22:01:09,954 INFO [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] regionserver.HRegion(1006): Closed test,ttt,1369432820320.f53d942877419dd243e3fb7e46c2cf97. 2013-05-24 22:01:09,955 DEBUG [RS_CLOSE_REGION-kiyo.gq1.ygridcore.net,42690,1369432806911-2] handler.CloseRegionHandler(177): Closed region test,ttt,1369432820320.f53d942877419dd243e3fb7e46c2cf97. 2013-05-24 22:01:10,010 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911] regionserver.HRegionServer(912): stopping server kiyo.gq1.ygridcore.net,42690,1369432806911; all regions closed. 2013-05-24 22:01:10,010 INFO [RS_OPEN_META-kiyo.gq1.ygridcore.net,42690,1369432806911-0.logSyncer] wal.FSHLog$LogSyncer(967): RS_OPEN_META-kiyo.gq1.ygridcore.net,42690,1369432806911-0.logSyncer exiting 2013-05-24 22:01:10,010 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911] wal.FSHLog(809): closing hlog writer in hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:01:10,012 INFO [IPC Server handler 3 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59993 is added to blk_-309222136226719791_1033{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:53333|RBW], ReplicaUnderConstruction[127.0.0.1:59993|RBW]]} size 0 2013-05-24 22:01:10,013 INFO [IPC Server handler 2 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53333 is added to blk_-309222136226719791_1033{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:53333|RBW], ReplicaUnderConstruction[127.0.0.1:59993|RBW]]} size 0 2013-05-24 22:01:10,072 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.logSyncer] wal.FSHLog$LogSyncer(967): RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.logSyncer exiting 2013-05-24 22:01:10,072 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911] wal.FSHLog(809): closing hlog writer in hdfs://localhost:54096/user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:01:10,074 INFO [IPC Server handler 4 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59993 is added to blk_-2639188558578359449_1046{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:53333|RBW], ReplicaUnderConstruction[127.0.0.1:59993|RBW]]} size 0 2013-05-24 22:01:10,075 INFO [IPC Server handler 9 on 54096] blockmanagement.BlockManager(2200): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53333 is added to blk_-2639188558578359449_1046 size 8224 2013-05-24 22:01:10,380 INFO [Master:0;kiyo.gq1.ygridcore.net,49101,1369432806564] master.ServerManager(448): Waiting on regionserver(s) to go down kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:01:11,131 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911] wal.FSHLog(768): Moved 10 log files to /user/hortonzy/hbase/.oldlogs 2013-05-24 22:01:11,261 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911] regionserver.Leases(124): RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911 closing leases 2013-05-24 22:01:11,262 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911] regionserver.Leases(131): RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911 closed leases 2013-05-24 22:01:11,262 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911] regionserver.ReplicationSource(761): Closing source 2 because: Region server is closing 2013-05-24 22:01:11,262 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911] client.HConnectionManager$HConnectionImplementation(1640): Closing zookeeper sessionid=0x13ed88e2cdf000e 2013-05-24 22:01:11,337 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:42690-0x13ed88e2cdf0006 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/rs/kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:01:11,337 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/rs/kiyo.gq1.ygridcore.net,42690,1369432806911 2013-05-24 22:01:11,337 INFO [pool-1-thread-1-EventThread] zookeeper.RegionServerTracker(94): RegionServer ephemeral node deleted, processing expiration [kiyo.gq1.ygridcore.net,42690,1369432806911] 2013-05-24 22:01:11,337 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:42690-0x13ed88e2cdf0006 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/rs 2013-05-24 22:01:11,337 INFO [pool-1-thread-1-EventThread] master.ServerManager(495): Cluster shutdown set; kiyo.gq1.ygridcore.net,42690,1369432806911 expired; onlineServers=0 2013-05-24 22:01:11,337 DEBUG [Master:0;kiyo.gq1.ygridcore.net,49101,1369432806564] master.HMaster(1111): Stopping service threads 2013-05-24 22:01:11,337 INFO [pool-1-thread-1-EventThread] master.HMaster(2216): Cluster shutdown set; onlineServer=0 2013-05-24 22:01:11,338 DEBUG [pool-1-thread-1-EventThread] catalog.CatalogTracker(208): Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@15c8f644 2013-05-24 22:01:11,339 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:49101-0x13ed88e2cdf0004 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/rs 2013-05-24 22:01:11,366 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911] regionserver.HRegionServer(941): stopping server kiyo.gq1.ygridcore.net,42690,1369432806911; zookeeper connection closed. 2013-05-24 22:01:11,366 INFO [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911] regionserver.HRegionServer(944): RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911 exiting 2013-05-24 22:01:11,367 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@672d34a6] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(193): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@672d34a6 2013-05-24 22:01:11,367 INFO [pool-1-thread-1] util.JVMClusterUtil(309): Shutdown of 1 master(s) and 2 regionserver(s) complete 2013-05-24 22:01:11,386 INFO [Master:0;kiyo.gq1.ygridcore.net,49101,1369432806564] master.HMaster(587): HMaster main thread exiting 2013-05-24 22:01:11,389 INFO [pool-1-thread-1] zookeeper.MiniZooKeeperCluster(246): Shutdown MiniZK cluster with all ZK servers 2013-05-24 22:01:11,390 WARN [pool-1-thread-1] datanode.DirectoryScanner(289): DirectoryScanner: shutdown has been called 2013-05-24 22:01:11,393 INFO [pool-1-thread-1] log.Slf4jLog(67): Stopped SelectChannelConnector@localhost:0 2013-05-24 22:01:11,487 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): hconnection-0x63c9982e-0x13ed88e2cdf0019 Received ZooKeeper Event, type=None, state=Disconnected, path=null 2013-05-24 22:01:11,487 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(389): hconnection-0x63c9982e-0x13ed88e2cdf0019 Received Disconnected from ZooKeeper, ignoring 2013-05-24 22:01:11,487 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997-EventThread] zookeeper.ZooKeeperWatcher(307): connection to cluster: localhost:51411:/2-0x13ed88e2cdf000b Received ZooKeeper Event, type=None, state=Disconnected, path=null 2013-05-24 22:01:11,487 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): hconnection-0x3e152f4-0x13ed88e2cdf0001 Received ZooKeeper Event, type=None, state=Disconnected, path=null 2013-05-24 22:01:11,487 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): cluster2-0x13ed88e2cdf0003 Received ZooKeeper Event, type=None, state=Disconnected, path=null 2013-05-24 22:01:11,487 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(389): cluster2-0x13ed88e2cdf0003 Received Disconnected from ZooKeeper, ignoring 2013-05-24 22:01:11,487 DEBUG [RegionServer:1;kiyo.gq1.ygridcore.net,49701,1369432806997-EventThread] zookeeper.ZooKeeperWatcher(389): connection to cluster: localhost:51411:/2-0x13ed88e2cdf000b Received Disconnected from ZooKeeper, ignoring 2013-05-24 22:01:11,488 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): cluster1-0x13ed88e2cdf0000 Received ZooKeeper Event, type=None, state=Disconnected, path=null 2013-05-24 22:01:11,488 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(389): cluster1-0x13ed88e2cdf0000 Received Disconnected from ZooKeeper, ignoring 2013-05-24 22:01:11,487 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(389): hconnection-0x3e152f4-0x13ed88e2cdf0001 Received Disconnected from ZooKeeper, ignoring 2013-05-24 22:01:11,488 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.ZooKeeperWatcher(307): hconnection-0x1271ba-0x13ed88e2cdf0008 Received ZooKeeper Event, type=None, state=Disconnected, path=null 2013-05-24 22:01:11,488 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.ZooKeeperWatcher(389): hconnection-0x1271ba-0x13ed88e2cdf0008 Received Disconnected from ZooKeeper, ignoring 2013-05-24 22:01:11,488 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.ZooKeeperWatcher(307): connection to cluster: localhost:51411:/2-0x13ed88e2cdf000c Received ZooKeeper Event, type=None, state=Disconnected, path=null 2013-05-24 22:01:11,488 DEBUG [Master:0;kiyo.gq1.ygridcore.net,49101,1369432806564-EventThread] zookeeper.ZooKeeperWatcher(307): hconnection-0x4b0d78ec-0x13ed88e2cdf0007 Received ZooKeeper Event, type=None, state=Disconnected, path=null 2013-05-24 22:01:11,488 DEBUG [Master:0;kiyo.gq1.ygridcore.net,49101,1369432806564-EventThread] zookeeper.ZooKeeperWatcher(389): hconnection-0x4b0d78ec-0x13ed88e2cdf0007 Received Disconnected from ZooKeeper, ignoring 2013-05-24 22:01:11,488 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911-EventThread] zookeeper.ZooKeeperWatcher(389): connection to cluster: localhost:51411:/2-0x13ed88e2cdf000c Received Disconnected from ZooKeeper, ignoring 2013-05-24 22:01:11,488 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): Replication Admin-0x13ed88e2cdf0002 Received ZooKeeper Event, type=None, state=Disconnected, path=null 2013-05-24 22:01:11,489 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(389): Replication Admin-0x13ed88e2cdf0002 Received Disconnected from ZooKeeper, ignoring 2013-05-24 22:01:11,497 WARN [DataNode: [file:/homes/hortonzy/trunk/hbase-server/target/test-data/0be71c72-d7a8-4ebf-8710-33a58f98200c/dfscluster_e74483c2-5e62-427d-82be-23dadbbd1b61/dfs/data/data3,file:/homes/hortonzy/trunk/hbase-server/target/test-data/0be71c72-d7a8-4ebf-8710-33a58f98200c/dfscluster_e74483c2-5e62-427d-82be-23dadbbd1b61/dfs/data/data4] heartbeating to localhost/127.0.0.1:54096] datanode.BPServiceActor(575): BPOfferService for Block pool BP-1702562054-68.142.244.61-1369432800011 (storage id DS-1797281922-68.142.244.61-59993-1369432804974) service to localhost/127.0.0.1:54096 interrupted 2013-05-24 22:01:11,497 WARN [DataNode: [file:/homes/hortonzy/trunk/hbase-server/target/test-data/0be71c72-d7a8-4ebf-8710-33a58f98200c/dfscluster_e74483c2-5e62-427d-82be-23dadbbd1b61/dfs/data/data3,file:/homes/hortonzy/trunk/hbase-server/target/test-data/0be71c72-d7a8-4ebf-8710-33a58f98200c/dfscluster_e74483c2-5e62-427d-82be-23dadbbd1b61/dfs/data/data4] heartbeating to localhost/127.0.0.1:54096] datanode.BPServiceActor(685): Ending block pool service for: Block pool BP-1702562054-68.142.244.61-1369432800011 (storage id DS-1797281922-68.142.244.61-59993-1369432804974) service to localhost/127.0.0.1:54096 2013-05-24 22:01:11,498 WARN [pool-1-thread-1] datanode.DirectoryScanner(289): DirectoryScanner: shutdown has been called 2013-05-24 22:01:11,501 INFO [pool-1-thread-1] log.Slf4jLog(67): Stopped SelectChannelConnector@localhost:0 2013-05-24 22:01:11,605 WARN [DataNode: [file:/homes/hortonzy/trunk/hbase-server/target/test-data/0be71c72-d7a8-4ebf-8710-33a58f98200c/dfscluster_e74483c2-5e62-427d-82be-23dadbbd1b61/dfs/data/data1,file:/homes/hortonzy/trunk/hbase-server/target/test-data/0be71c72-d7a8-4ebf-8710-33a58f98200c/dfscluster_e74483c2-5e62-427d-82be-23dadbbd1b61/dfs/data/data2] heartbeating to localhost/127.0.0.1:54096] datanode.BPServiceActor(575): BPOfferService for Block pool BP-1702562054-68.142.244.61-1369432800011 (storage id DS-574199218-68.142.244.61-53333-1369432804846) service to localhost/127.0.0.1:54096 interrupted 2013-05-24 22:01:11,605 WARN [DataNode: [file:/homes/hortonzy/trunk/hbase-server/target/test-data/0be71c72-d7a8-4ebf-8710-33a58f98200c/dfscluster_e74483c2-5e62-427d-82be-23dadbbd1b61/dfs/data/data1,file:/homes/hortonzy/trunk/hbase-server/target/test-data/0be71c72-d7a8-4ebf-8710-33a58f98200c/dfscluster_e74483c2-5e62-427d-82be-23dadbbd1b61/dfs/data/data2] heartbeating to localhost/127.0.0.1:54096] datanode.BPServiceActor(685): Ending block pool service for: Block pool BP-1702562054-68.142.244.61-1369432800011 (storage id DS-574199218-68.142.244.61-53333-1369432804846) service to localhost/127.0.0.1:54096 2013-05-24 22:01:11,620 WARN [org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@46b113c7] blockmanagement.BlockManager$ReplicationMonitor(3110): ReplicationMonitor thread received InterruptedException. java.lang.InterruptedException: sleep interrupted at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3108) at java.lang.Thread.run(Thread.java:662) 2013-05-24 22:01:11,621 WARN [org.apache.hadoop.hdfs.server.blockmanagement.DecommissionManager$Monitor@66952905] blockmanagement.DecommissionManager$Monitor(78): Monitor interrupted: java.lang.InterruptedException: sleep interrupted 2013-05-24 22:01:11,630 INFO [pool-1-thread-1] log.Slf4jLog(67): Stopped SelectChannelConnector@localhost:0 2013-05-24 22:01:11,754 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(920): Minicluster is down