[2013-01-11 09:40:36,740] INFO Starting mirror maker (kafka.tools.MirrorMaker$) [2013-01-11 09:40:36,796] INFO Verifying properties (kafka.utils.VerifiableProperties) [2013-01-11 09:40:36,797] INFO Property broker.list is overridden to 10.0.0.2:9092 (kafka.utils.VerifiableProperties) [2013-01-11 09:40:36,797] INFO Property queue.size is overridden to 100000 (kafka.utils.VerifiableProperties) [2013-01-11 09:40:36,797] INFO Property producer.type is overridden to async (kafka.utils.VerifiableProperties) [2013-01-11 09:40:36,798] INFO Property batch.size is overridden to 20000 (kafka.utils.VerifiableProperties) [2013-01-11 09:40:36,798] INFO Property queue.enqueueTimeout.ms is overridden to -1 (kafka.utils.VerifiableProperties) [2013-01-11 09:40:36,864] INFO Verifying properties (kafka.utils.VerifiableProperties) [2013-01-11 09:40:36,864] INFO Property zk.connect is overridden to 10.0.0.1:2181 (kafka.utils.VerifiableProperties) [2013-01-11 09:40:36,864] INFO Property socket.buffersize is overridden to 26214400 (kafka.utils.VerifiableProperties) [2013-01-11 09:40:36,865] INFO Property consumer.timeout.ms is overridden to -1 (kafka.utils.VerifiableProperties) [2013-01-11 09:40:36,865] INFO Property groupid is overridden to KafkaMirror (kafka.utils.VerifiableProperties) [2013-01-11 09:40:36,865] INFO Property queuedchunks.max is overridden to 10000 (kafka.utils.VerifiableProperties) [2013-01-11 09:40:36,865] INFO Property shallowiterator.enable is overridden to true (kafka.utils.VerifiableProperties) [2013-01-11 09:40:36,865] INFO Property zk.connectiontimeout.ms is overridden to 1000000 (kafka.utils.VerifiableProperties) [2013-01-11 09:40:36,866] INFO Property fetch.size is overridden to 25485760 (kafka.utils.VerifiableProperties) [2013-01-11 09:40:36,866] INFO Property max.fetch.wait.ms is overridden to 1000 (kafka.utils.VerifiableProperties) [2013-01-11 09:40:36,866] INFO Property min.fetch.bytes is overridden to 1000 (kafka.utils.VerifiableProperties) [2013-01-11 09:40:36,866] INFO Property refresh.leader.backoff.ms is overridden to 600000 (kafka.utils.VerifiableProperties) [2013-01-11 09:40:36,875] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], Connecting to zookeeper instance at 10.0.0.1:2181 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:36,884] DEBUG Creating new ZookKeeper instance to connect to 10.0.0.1:2181. (org.I0Itec.zkclient.ZkConnection) [2013-01-11 09:40:36,884] INFO Starting ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread) [2013-01-11 09:40:36,891] INFO Client environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 GMT (org.apache.zookeeper.ZooKeeper) [2013-01-11 09:40:36,891] INFO Client environment:host.name=mirror-01 (org.apache.zookeeper.ZooKeeper) [2013-01-11 09:40:36,891] INFO Client environment:java.version=1.6.0_31 (org.apache.zookeeper.ZooKeeper) [2013-01-11 09:40:36,891] INFO Client environment:java.vendor=Sun Microsystems Inc. (org.apache.zookeeper.ZooKeeper) [2013-01-11 09:40:36,891] INFO Client environment:java.home=/usr/java/jdk1.6.0_31/jre (org.apache.zookeeper.ZooKeeper) [2013-01-11 09:40:36,891] INFO Client environment:java.class.path=:/opt/kafka-0.8/bin/../project/boot/scala-2.8.0/lib/scala-compiler.jar:/opt/kafka-0.8/bin/../project/boot/scala-2.8.0/lib/scala-library.jar:/opt/kafka-0.8/bin/../core/target/scala_2.8.0/core-kafka_2.8.0-0.8.0-SNAPSHOT.jar:/opt/kafka-0.8/bin/../core/target/scala_2.8.0/core-kafka_2.8.0-0.8.0-SNAPSHOT-test.jar:/opt/kafka-0.8/bin/../core/lib/metrics-annotation-3.0.0-c0c8be71.jar:/opt/kafka-0.8/bin/../core/lib/metrics-core-3.0.0-c0c8be71.jar:/opt/kafka-0.8/bin/../core/lib/zkclient-20120522.jar:/opt/kafka-0.8/bin/../perf/target/scala_2.8.0/kafka-perf-0.8.0-SNAPSHOT.jar:/opt/kafka-0.8/bin/../core/lib_managed/scala_2.8.0/compile/jopt-simple-3.2.jar:/opt/kafka-0.8/bin/../core/lib_managed/scala_2.8.0/compile/log4j-1.2.15.jar:/opt/kafka-0.8/bin/../core/lib_managed/scala_2.8.0/compile/slf4j-api-1.7.2.jar:/opt/kafka-0.8/bin/../core/lib_managed/scala_2.8.0/compile/slf4j-simple-1.7.2.jar:/opt/kafka-0.8/bin/../core/lib_managed/scala_2.8.0/compile/snappy-java-1.0.4.1.jar:/opt/kafka-0.8/bin/../core/lib_managed/scala_2.8.0/compile/zookeeper-3.3.4.jar (org.apache.zookeeper.ZooKeeper) [2013-01-11 09:40:36,891] INFO Client environment:java.library.path=/usr/java/jdk1.6.0_31/jre/lib/amd64/server:/usr/java/jdk1.6.0_31/jre/lib/amd64:/usr/java/jdk1.6.0_31/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper) [2013-01-11 09:40:36,891] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper) [2013-01-11 09:40:36,891] INFO Client environment:java.compiler= (org.apache.zookeeper.ZooKeeper) [2013-01-11 09:40:36,891] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper) [2013-01-11 09:40:36,891] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper) [2013-01-11 09:40:36,891] INFO Client environment:os.version=2.6.18-274.12.1.el5.centos.plus (org.apache.zookeeper.ZooKeeper) [2013-01-11 09:40:36,891] INFO Client environment:user.name=m.brugidou (org.apache.zookeeper.ZooKeeper) [2013-01-11 09:40:36,891] INFO Client environment:user.home=/home/m.brugidou (org.apache.zookeeper.ZooKeeper) [2013-01-11 09:40:36,891] INFO Client environment:user.dir=/home/m.brugidou (org.apache.zookeeper.ZooKeeper) [2013-01-11 09:40:36,892] INFO Initiating client connection, connectString=10.0.0.1:2181 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient@3288df60 (org.apache.zookeeper.ZooKeeper) [2013-01-11 09:40:36,894] DEBUG zookeeper.disableAutoWatchReset is false (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:36,908] DEBUG Awaiting connection to Zookeeper server (org.I0Itec.zkclient.ZkClient) [2013-01-11 09:40:36,908] INFO Opening socket connection to server /10.0.0.1:2181 (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:36,908] DEBUG Waiting for keeper state SyncConnected (org.I0Itec.zkclient.ZkClient) [2013-01-11 09:40:37,058] INFO Socket connection established to broker-01/10.0.0.1:2181, initiating session (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:37,060] DEBUG Session establishment request sent on broker-01/10.0.0.1:2181 (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:37,207] INFO Session establishment complete on server broker-01/10.0.0.1:2181, sessionid = 0x13c012a599601aa, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:37,209] DEBUG Received event: WatchedEvent state:SyncConnected type:None path:null (org.I0Itec.zkclient.ZkClient) [2013-01-11 09:40:37,209] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient) [2013-01-11 09:40:37,209] DEBUG Leaving process event (org.I0Itec.zkclient.ZkClient) [2013-01-11 09:40:37,209] DEBUG State is SyncConnected (org.I0Itec.zkclient.ZkClient) [2013-01-11 09:40:37,214] INFO [KafkaMirror_mirror-01-1357893636873-1b343568-leader-finder-thread], Starting (kafka.consumer.ConsumerFetcherManager$$anon$1) [2013-01-11 09:40:37,215] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], starting auto committer every 60000 ms (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:37,387] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 1,8 replyHeader:: 1,420841,0 request:: '/brokers/topics,F response:: v{'topic-23,'topic-5,'topic-1,'topic-7,'topic-14,'topic-6,'topic-8,'topic-92,'topic-9al,'topic-3,'topic-2,'topic-15,'topic-19,'dummy,'topic-24,'topic-20,'topic-16,'topic-18,'topic-21,'topic-3,'topic-22,'topic-25,'topic-17,'topic-9} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:37,389] DEBUG topic-23 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:37,389] DEBUG topic-5 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:37,390] DEBUG topic-1 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:37,390] DEBUG topic-7 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:37,390] DEBUG topic-14 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:37,390] DEBUG topic-6 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:37,391] DEBUG topic-8 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:37,391] DEBUG topic-92 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:37,391] DEBUG topic-9al allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:37,392] DEBUG topic-3 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:37,392] DEBUG topic-2 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:37,392] DEBUG topic-15 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:37,392] DEBUG topic-19 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:37,392] DEBUG dummy allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:37,393] DEBUG topic-24 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:37,393] DEBUG topic-20 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:37,393] DEBUG topic-16 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:37,393] DEBUG topic-18 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:37,393] DEBUG topic-21 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:37,394] DEBUG topic-3 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:37,394] DEBUG topic-22 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:37,394] DEBUG topic-25 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:37,394] DEBUG topic-17 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:37,394] DEBUG topic-9 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:37,399] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], begin registering consumer KafkaMirror_mirror-01-1357893636873-1b343568 in ZK (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:37,549] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 2,1 replyHeader:: 2,420842,0 request:: '/consumers/KafkaMirror/ids/KafkaMirror_mirror-01-1357893636873-1b343568,#2131217864756d6d7978,v{s{31,s{'world,'anyone}}},1 response:: '/consumers/KafkaMirror/ids/KafkaMirror_mirror-01-1357893636873-1b343568 (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:37,549] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], end registering consumer KafkaMirror_mirror-01-1357893636873-1b343568 in ZK (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:37,553] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], starting watcher executor thread for consumer KafkaMirror_mirror-01-1357893636873-1b343568 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:37,698] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 3,8 replyHeader:: 3,420842,0 request:: '/brokers/topics,F response:: v{'topic-23,'topic-5,'topic-1,'topic-7,'topic-14,'topic-6,'topic-8,'topic-92,'topic-9al,'topic-3,'topic-2,'topic-15,'topic-19,'dummy,'topic-24,'topic-20,'topic-16,'topic-18,'topic-21,'topic-3,'topic-22,'topic-25,'topic-17,'topic-9} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:37,699] DEBUG topic-23 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:37,699] DEBUG topic-5 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:37,699] DEBUG topic-1 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:37,700] DEBUG topic-7 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:37,700] DEBUG topic-14 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:37,700] DEBUG topic-6 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:37,700] DEBUG topic-8 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:37,700] DEBUG topic-92 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:37,701] DEBUG topic-9al allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:37,701] DEBUG topic-3 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:37,701] DEBUG topic-2 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:37,701] DEBUG topic-15 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:37,701] DEBUG topic-19 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:37,702] DEBUG dummy allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:37,702] DEBUG topic-24 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:37,702] DEBUG topic-20 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:37,702] DEBUG topic-16 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:37,702] DEBUG topic-18 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:37,703] DEBUG topic-21 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:37,703] DEBUG topic-3 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:37,703] DEBUG topic-22 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:37,703] DEBUG topic-25 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:37,703] DEBUG topic-17 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:37,704] DEBUG topic-9 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:37,751] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], adding topic topic-19 and 1 streams to map. (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:37,751] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], adding topic topic-3 and 1 streams to map. (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:37,752] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], adding topic topic-21 and 1 streams to map. (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:37,752] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], adding topic topic-2 and 1 streams to map. (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:37,752] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], adding topic topic-25 and 1 streams to map. (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:37,752] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], adding topic topic-6 and 1 streams to map. (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:37,753] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], adding topic topic-14 and 1 streams to map. (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:37,753] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], adding topic topic-92 and 1 streams to map. (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:37,753] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], adding topic topic-23 and 1 streams to map. (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:37,753] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], adding topic topic-7 and 1 streams to map. (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:37,753] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], adding topic topic-3 and 1 streams to map. (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:37,754] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], adding topic dummy and 1 streams to map. (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:37,754] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], adding topic topic-18 and 1 streams to map. (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:37,754] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], adding topic topic-17 and 1 streams to map. (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:37,754] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], adding topic topic-5 and 1 streams to map. (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:37,754] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], adding topic topic-15 and 1 streams to map. (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:37,755] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], adding topic topic-22 and 1 streams to map. (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:37,755] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], adding topic topic-16 and 1 streams to map. (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:37,755] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], adding topic topic-9al and 1 streams to map. (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:37,755] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], adding topic topic-20 and 1 streams to map. (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:37,756] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], adding topic topic-9 and 1 streams to map. (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:37,756] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], adding topic topic-24 and 1 streams to map. (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:37,756] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], adding topic topic-8 and 1 streams to map. (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:37,756] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], adding topic topic-1 and 1 streams to map. (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:37,906] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 4,3 replyHeader:: 4,420842,0 request:: '/consumers/KafkaMirror/ids,T response:: s{258,258,1357229306874,1357229306874,0,93,0,0,0,1,420842} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:38,051] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 5,8 replyHeader:: 5,420842,0 request:: '/consumers/KafkaMirror/ids,T response:: v{'KafkaMirror_mirror-01-1357893636873-1b343568} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:38,197] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 6,3 replyHeader:: 6,420842,0 request:: '/brokers/topics/topic-1,T response:: s{1691,1691,1357230876706,1357230876706,0,1,0,0,22,1,1917} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:38,342] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 7,8 replyHeader:: 7,420842,0 request:: '/brokers/topics/topic-1,T response:: v{'partitions} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:38,487] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 8,3 replyHeader:: 8,420842,0 request:: '/brokers/topics/topic-8,T response:: s{1703,1703,1357230880145,1357230880145,0,1,0,0,22,1,1827} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:38,632] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 9,8 replyHeader:: 9,420842,0 request:: '/brokers/topics/topic-8,T response:: v{'partitions} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:38,777] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 10,3 replyHeader:: 10,420842,0 request:: '/brokers/topics/topic-24,T response:: s{1659,1659,1357230797732,1357230797732,0,1,0,0,79,1,1663} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:38,922] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 11,8 replyHeader:: 11,420842,0 request:: '/brokers/topics/topic-24,T response:: v{'partitions} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:39,068] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 12,3 replyHeader:: 12,420842,0 request:: '/brokers/topics/topic-9,T response:: s{305740,305740,1357699009528,1357699009528,0,1,0,0,22,1,305743} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:39,213] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 13,8 replyHeader:: 13,420842,0 request:: '/brokers/topics/topic-9,T response:: v{'partitions} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:39,358] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 14,3 replyHeader:: 14,420842,0 request:: '/brokers/topics/topic-20,T response:: s{200840,200840,1357565032025,1357565032025,0,1,0,0,22,1,200844} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:39,503] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 15,8 replyHeader:: 15,420842,0 request:: '/brokers/topics/topic-20,T response:: v{'partitions} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:39,648] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 16,3 replyHeader:: 16,420842,0 request:: '/brokers/topics/topic-9al,T response:: s{227867,227867,1357596915974,1357596915974,0,1,0,0,22,1,227870} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:39,793] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 17,8 replyHeader:: 17,420842,0 request:: '/brokers/topics/topic-9al,T response:: v{'partitions} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:39,938] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 18,3 replyHeader:: 18,420842,0 request:: '/brokers/topics/topic-16,T response:: s{1712,1712,1357230882648,1357230882648,0,1,0,0,22,1,1794} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:40,083] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 19,8 replyHeader:: 19,420842,0 request:: '/brokers/topics/topic-16,T response:: v{'partitions} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:40,229] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 20,3 replyHeader:: 20,420842,0 request:: '/brokers/topics/topic-22,T response:: s{1724,1724,1357230886150,1357230886150,0,1,0,0,22,1,1789} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:40,374] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 21,8 replyHeader:: 21,420842,0 request:: '/brokers/topics/topic-22,T response:: v{'partitions} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:40,519] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 22,3 replyHeader:: 22,420842,0 request:: '/brokers/topics/topic-15,T response:: s{200913,200913,1357566275007,1357566275007,0,1,0,0,22,1,200916} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:40,664] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 23,8 replyHeader:: 23,420842,0 request:: '/brokers/topics/topic-15,T response:: v{'partitions} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:40,809] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 24,3 replyHeader:: 24,420842,0 request:: '/brokers/topics/topic-5,T response:: s{200875,200875,1357565193670,1357565193670,0,1,0,0,22,1,200879} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:40,954] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 25,8 replyHeader:: 25,420842,0 request:: '/brokers/topics/topic-5,T response:: v{'partitions} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:41,099] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 26,3 replyHeader:: 26,420842,0 request:: '/brokers/topics/topic-17,T response:: s{1715,1715,1357230883495,1357230883495,0,1,0,0,22,1,1862} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:41,244] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 27,8 replyHeader:: 27,420842,0 request:: '/brokers/topics/topic-17,T response:: v{'partitions} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:41,389] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 28,3 replyHeader:: 28,420842,0 request:: '/brokers/topics/dummy,T response:: s{200925,302028,1357566917729,1357665172205,2,1,0,0,109,1,200928} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:41,534] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 29,8 replyHeader:: 29,420842,0 request:: '/brokers/topics/dummy,T response:: v{'partitions} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:41,679] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 30,3 replyHeader:: 30,420842,0 request:: '/brokers/topics/topic-18,T response:: s{200901,200901,1357565538836,1357565538836,0,1,0,0,22,1,200904} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:41,824] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 31,8 replyHeader:: 31,420842,0 request:: '/brokers/topics/topic-18,T response:: v{'partitions} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:41,855] DEBUG 5004 ms elapsed. Queue time reached. Sending.. (kafka.producer.async.ProducerSendThread) [2013-01-11 09:40:41,856] DEBUG Handling 0 events (kafka.producer.async.ProducerSendThread) [2013-01-11 09:40:41,970] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 32,3 replyHeader:: 32,420842,0 request:: '/brokers/topics/topic-3,T response:: s{207326,207326,1357575051642,1357575051642,0,1,0,0,22,1,207329} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:42,115] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 33,8 replyHeader:: 33,420842,0 request:: '/brokers/topics/topic-3,T response:: v{'partitions} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:42,260] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 34,3 replyHeader:: 34,420842,0 request:: '/brokers/topics/topic-7,T response:: s{1700,1700,1357230879291,1357230879291,0,1,0,0,22,1,1832} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:42,405] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 35,8 replyHeader:: 35,420842,0 request:: '/brokers/topics/topic-7,T response:: v{'partitions} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:42,550] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 36,3 replyHeader:: 36,420842,0 request:: '/brokers/topics/topic-23,T response:: s{1706,214303,1357230880969,1357583155042,1,1,0,0,37,1,1912} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:42,695] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 37,8 replyHeader:: 37,420842,0 request:: '/brokers/topics/topic-23,T response:: v{'partitions} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:42,840] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 38,3 replyHeader:: 38,420842,0 request:: '/brokers/topics/topic-14,T response:: s{1709,1709,1357230881819,1357230881819,0,1,0,0,22,1,1804} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:42,985] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 39,8 replyHeader:: 39,420842,0 request:: '/brokers/topics/topic-14,T response:: v{'partitions} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:43,130] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 40,3 replyHeader:: 40,420842,0 request:: '/brokers/topics/topic-92,T response:: s{420724,420724,1357893142883,1357893142883,0,1,0,0,22,1,420727} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:43,275] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 41,8 replyHeader:: 41,420842,0 request:: '/brokers/topics/topic-92,T response:: v{'partitions} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:43,420] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 42,3 replyHeader:: 42,420842,0 request:: '/brokers/topics/topic-6,T response:: s{1697,1697,1357230878431,1357230878431,0,1,0,0,22,1,1922} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:43,565] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 43,8 replyHeader:: 43,420842,0 request:: '/brokers/topics/topic-6,T response:: v{'partitions} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:43,710] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 44,3 replyHeader:: 44,420842,0 request:: '/brokers/topics/topic-25,T response:: s{200907,200907,1357565709429,1357565709429,0,1,0,0,22,1,200910} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:43,855] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 45,8 replyHeader:: 45,420842,0 request:: '/brokers/topics/topic-25,T response:: v{'partitions} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:44,000] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 46,3 replyHeader:: 46,420842,0 request:: '/brokers/topics/topic-21,T response:: s{1721,1721,1357230885294,1357230885294,0,1,0,0,22,1,1872} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:44,145] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 47,8 replyHeader:: 47,420842,0 request:: '/brokers/topics/topic-21,T response:: v{'partitions} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:44,290] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 48,3 replyHeader:: 48,420842,0 request:: '/brokers/topics/topic-2,T response:: s{1694,1694,1357230877589,1357230877589,0,1,0,0,22,1,1852} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:44,435] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 49,8 replyHeader:: 49,420842,0 request:: '/brokers/topics/topic-2,T response:: v{'partitions} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:44,581] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 50,3 replyHeader:: 50,420842,0 request:: '/brokers/topics/topic-3,T response:: s{1624,291192,1357230748551,1357655430103,1,1,0,0,193,1,1628} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:44,726] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 51,8 replyHeader:: 51,420842,0 request:: '/brokers/topics/topic-3,T response:: v{'partitions} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:44,871] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 52,3 replyHeader:: 52,420842,0 request:: '/brokers/topics/topic-19,T response:: s{1718,1718,1357230884439,1357230884439,0,1,0,0,22,1,1907} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:45,016] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 53,8 replyHeader:: 53,420842,0 request:: '/brokers/topics/topic-19,T response:: v{'partitions} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:45,018] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], begin rebalancing consumer KafkaMirror_mirror-01-1357893636873-1b343568 try #0 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:45,164] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 54,8 replyHeader:: 54,420842,0 request:: '/brokers/ids,F response:: v{'174984962} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:45,311] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 55,4 replyHeader:: 55,420842,0 request:: '/brokers/ids/174984962,F response:: #6c6f672d6c3131302d31352d322e63726974656f2e70726f643a393039323a39393939,s{302210,302210,1357665300415,1357665300415,0,0,0,88947374043824448,35,0,302210} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:45,460] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 56,4 replyHeader:: 56,420842,0 request:: '/consumers/KafkaMirror/ids/KafkaMirror_mirror-01-1357893636873-1b343568,F response:: #2131217864756d6d7978,s{420842,420842,1357893637481,1357893637481,0,0,0,88947374043824554,10,0,420842} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:45,461] INFO Constructing topic count for KafkaMirror_mirror-01-1357893636873-1b343568 from !1!xdummyx using !(\p{Digit}+)!(.*) as pattern. (kafka.consumer.TopicCount$) [2013-01-11 09:40:45,607] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 57,8 replyHeader:: 57,420842,0 request:: '/brokers/topics,F response:: v{'topic-23,'topic-5,'topic-1,'topic-7,'topic-14,'topic-6,'topic-8,'topic-92,'topic-9al,'topic-3,'topic-2,'topic-15,'topic-19,'dummy,'topic-24,'topic-20,'topic-16,'topic-18,'topic-21,'topic-3,'topic-22,'topic-25,'topic-17,'topic-9} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:45,608] DEBUG topic-23 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:45,608] DEBUG topic-5 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:45,608] DEBUG topic-1 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:45,608] DEBUG topic-7 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:45,609] DEBUG topic-14 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:45,609] DEBUG topic-6 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:45,609] DEBUG topic-8 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:45,609] DEBUG topic-92 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:45,609] DEBUG topic-9al allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:45,610] DEBUG topic-3 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:45,610] DEBUG topic-2 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:45,610] DEBUG topic-15 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:45,610] DEBUG topic-19 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:45,610] DEBUG dummy allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:45,610] DEBUG topic-24 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:45,611] DEBUG topic-20 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:45,611] DEBUG topic-16 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:45,611] DEBUG topic-18 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:45,611] DEBUG topic-21 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:45,611] DEBUG topic-3 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:45,612] DEBUG topic-22 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:45,612] DEBUG topic-25 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:45,612] DEBUG topic-17 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:45,612] DEBUG topic-9 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:45,758] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 58,8 replyHeader:: 58,420842,0 request:: '/consumers/KafkaMirror/ids,T response:: v{'KafkaMirror_mirror-01-1357893636873-1b343568} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:45,904] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 59,4 replyHeader:: 59,420842,0 request:: '/consumers/KafkaMirror/ids/KafkaMirror_mirror-01-1357893636873-1b343568,F response:: #2131217864756d6d7978,s{420842,420842,1357893637481,1357893637481,0,0,0,88947374043824554,10,0,420842} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:45,904] INFO Constructing topic count for KafkaMirror_mirror-01-1357893636873-1b343568 from !1!xdummyx using !(\p{Digit}+)!(.*) as pattern. (kafka.consumer.TopicCount$) [2013-01-11 09:40:46,050] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 60,8 replyHeader:: 60,420842,0 request:: '/brokers/topics,F response:: v{'topic-23,'topic-5,'topic-1,'topic-7,'topic-14,'topic-6,'topic-8,'topic-92,'topic-9al,'topic-3,'topic-2,'topic-15,'topic-19,'dummy,'topic-24,'topic-20,'topic-16,'topic-18,'topic-21,'topic-3,'topic-22,'topic-25,'topic-17,'topic-9} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:46,050] DEBUG topic-23 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:46,050] DEBUG topic-5 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:46,051] DEBUG topic-1 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:46,051] DEBUG topic-7 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:46,051] DEBUG topic-14 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:46,051] DEBUG topic-6 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:46,051] DEBUG topic-8 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:46,051] DEBUG topic-92 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:46,052] DEBUG topic-9al allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:46,052] DEBUG topic-3 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:46,052] DEBUG topic-2 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:46,052] DEBUG topic-15 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:46,052] DEBUG topic-19 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:46,053] DEBUG dummy allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:46,053] DEBUG topic-24 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:46,053] DEBUG topic-20 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:46,053] DEBUG topic-16 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:46,053] DEBUG topic-18 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:46,054] DEBUG topic-21 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:46,054] DEBUG topic-3 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:46,054] DEBUG topic-22 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:46,054] DEBUG topic-25 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:46,054] DEBUG topic-17 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:46,054] DEBUG topic-9 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:46,211] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 61,8 replyHeader:: 61,420842,0 request:: '/brokers/ids,F response:: v{'174984962} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:46,358] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 62,4 replyHeader:: 62,420842,0 request:: '/brokers/ids/174984962,F response:: #6c6f672d6c3131302d31352d322e63726974656f2e70726f643a393039323a39393939,s{302210,302210,1357665300415,1357665300415,0,0,0,88947374043824448,35,0,302210} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:46,360] INFO Verifying properties (kafka.utils.VerifiableProperties) [2013-01-11 09:40:46,361] INFO Property broker.list is overridden to broker-01:9092 (kafka.utils.VerifiableProperties) [2013-01-11 09:40:46,361] INFO Property clientid is overridden to KafkaMirror (kafka.utils.VerifiableProperties) [2013-01-11 09:40:46,384] INFO Fetching metadata for topic Set(topic-1, topic-8, topic-24, topic-9, topic-20, topic-9al, topic-16, topic-22, topic-15, topic-5, topic-17, dummy, topic-18, topic-3, topic-7, topic-23, topic-14, topic-92, topic-6, topic-25, topic-21, topic-2, topic-3, topic-19) (kafka.client.ClientUtils$) [2013-01-11 09:40:46,534] DEBUG Created socket with SO_TIMEOUT = 1500 (requested 1500), SO_RCVBUF = 43690 (requested -1), SO_SNDBUF = 102400 (requested 102400). (kafka.network.BlockingChannel) [2013-01-11 09:40:46,534] INFO Connected to broker-01:9092 for producing (kafka.producer.SyncProducer) [2013-01-11 09:40:46,808] INFO Disconnecting from broker-01:9092 (kafka.producer.SyncProducer) [2013-01-11 09:40:46,818] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], Cleared all relevant queues for this fetcher (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:46,819] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:46,819] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:46,819] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:46,819] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:46,819] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:46,820] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:46,820] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:46,820] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:46,820] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:46,820] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:46,820] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:46,820] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:46,820] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:46,820] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:46,821] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:46,821] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:46,821] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:46,821] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:46,821] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:46,821] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:46,821] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:46,821] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:46,821] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:46,822] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:46,822] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], Cleared the data chunks in all the consumer message iterators (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:46,822] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], Committing all offsets after clearing the fetcher queues (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:46,823] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], Releasing partition ownership (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:46,825] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], Consumer KafkaMirror_mirror-01-1357893636873-1b343568 rebalancing the following partitions: ArrayBuffer(0) for topic topic-1 with consumers: List(KafkaMirror_mirror-01-1357893636873-1b343568-0) (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:46,827] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], KafkaMirror_mirror-01-1357893636873-1b343568-0 attempting to claim partition 0 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:46,828] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], Leader for partition 0 for topic topic-1 is 174984962 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:46,859] DEBUG 5003 ms elapsed. Queue time reached. Sending.. (kafka.producer.async.ProducerSendThread) [2013-01-11 09:40:46,859] DEBUG Handling 0 events (kafka.producer.async.ProducerSendThread) [2013-01-11 09:40:46,973] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 63,4 replyHeader:: 63,420842,0 request:: '/consumers/KafkaMirror/offsets/topic-1/0,F response:: #3339393433343839,s{471,420757,1357229396250,1357893156979,8824,0,0,0,8,0,471} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:46,975] DEBUG initial consumer offset of topic-1:0: fetched offset = 39943489: consumed offset = 39943489 is 39943489 (kafka.consumer.PartitionTopicInfo) [2013-01-11 09:40:46,975] DEBUG initial fetch offset of topic-1:0: fetched offset = 39943489: consumed offset = 39943489 is 39943489 (kafka.consumer.PartitionTopicInfo) [2013-01-11 09:40:46,976] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], topic-1:0: fetched offset = 39943489: consumed offset = 39943489 selected new offset 39943489 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:46,976] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], Consumer KafkaMirror_mirror-01-1357893636873-1b343568 rebalancing the following partitions: ArrayBuffer(0) for topic topic-8 with consumers: List(KafkaMirror_mirror-01-1357893636873-1b343568-0) (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:46,976] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], KafkaMirror_mirror-01-1357893636873-1b343568-0 attempting to claim partition 0 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:46,977] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], Leader for partition 0 for topic topic-8 is 174984962 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:47,122] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 64,4 replyHeader:: 64,420842,0 request:: '/consumers/KafkaMirror/offsets/topic-8/0,F response:: #343431393339,s{474,420758,1357229396698,1357893157124,8824,0,0,0,6,0,474} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:47,122] DEBUG initial consumer offset of topic-8:0: fetched offset = 441939: consumed offset = 441939 is 441939 (kafka.consumer.PartitionTopicInfo) [2013-01-11 09:40:47,122] DEBUG initial fetch offset of topic-8:0: fetched offset = 441939: consumed offset = 441939 is 441939 (kafka.consumer.PartitionTopicInfo) [2013-01-11 09:40:47,122] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], topic-8:0: fetched offset = 441939: consumed offset = 441939 selected new offset 441939 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:47,122] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], Consumer KafkaMirror_mirror-01-1357893636873-1b343568 rebalancing the following partitions: ArrayBuffer(0, 1, 2, 3) for topic topic-24 with consumers: List(KafkaMirror_mirror-01-1357893636873-1b343568-0) (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:47,122] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], KafkaMirror_mirror-01-1357893636873-1b343568-0 attempting to claim partition 0 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:47,123] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], Leader for partition 0 for topic topic-24 is 174984962 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:47,267] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 65,4 replyHeader:: 65,420842,0 request:: '/consumers/KafkaMirror/offsets/topic-24/0,F response:: #3230393036373137,s{462,420751,1357229394938,1357893156104,8824,0,0,0,8,0,462} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:47,268] DEBUG initial consumer offset of topic-24:0: fetched offset = 20906717: consumed offset = 20906717 is 20906717 (kafka.consumer.PartitionTopicInfo) [2013-01-11 09:40:47,268] DEBUG initial fetch offset of topic-24:0: fetched offset = 20906717: consumed offset = 20906717 is 20906717 (kafka.consumer.PartitionTopicInfo) [2013-01-11 09:40:47,268] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], topic-24:0: fetched offset = 20906717: consumed offset = 20906717 selected new offset 20906717 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:47,268] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], KafkaMirror_mirror-01-1357893636873-1b343568-0 attempting to claim partition 1 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:47,268] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], Leader for partition 1 for topic topic-24 is 174984962 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:47,413] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 66,4 replyHeader:: 66,420842,0 request:: '/consumers/KafkaMirror/offsets/topic-24/1,F response:: #3230373639333336,s{13571,420753,1357301283528,1357893156396,8807,0,0,0,8,0,13571} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:47,413] DEBUG initial consumer offset of topic-24:1: fetched offset = 20769336: consumed offset = 20769336 is 20769336 (kafka.consumer.PartitionTopicInfo) [2013-01-11 09:40:47,413] DEBUG initial fetch offset of topic-24:1: fetched offset = 20769336: consumed offset = 20769336 is 20769336 (kafka.consumer.PartitionTopicInfo) [2013-01-11 09:40:47,413] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], topic-24:1: fetched offset = 20769336: consumed offset = 20769336 selected new offset 20769336 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:47,414] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], KafkaMirror_mirror-01-1357893636873-1b343568-0 attempting to claim partition 2 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:47,414] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], Leader for partition 2 for topic topic-24 is 174984962 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:47,559] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 67,4 replyHeader:: 67,420842,0 request:: '/consumers/KafkaMirror/offsets/topic-24/2,F response:: #3230373637383233,s{13568,420752,1357301283093,1357893156249,8807,0,0,0,8,0,13568} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:47,559] DEBUG initial consumer offset of topic-24:2: fetched offset = 20767823: consumed offset = 20767823 is 20767823 (kafka.consumer.PartitionTopicInfo) [2013-01-11 09:40:47,559] DEBUG initial fetch offset of topic-24:2: fetched offset = 20767823: consumed offset = 20767823 is 20767823 (kafka.consumer.PartitionTopicInfo) [2013-01-11 09:40:47,559] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], topic-24:2: fetched offset = 20767823: consumed offset = 20767823 selected new offset 20767823 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:47,559] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], KafkaMirror_mirror-01-1357893636873-1b343568-0 attempting to claim partition 3 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:47,560] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], Leader for partition 3 for topic topic-24 is 174984962 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:47,704] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 68,4 replyHeader:: 68,420842,0 request:: '/consumers/KafkaMirror/offsets/topic-24/3,F response:: #3230373639313638,s{13574,420754,1357301283963,1357893156541,8807,0,0,0,8,0,13574} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:47,705] DEBUG initial consumer offset of topic-24:3: fetched offset = 20769168: consumed offset = 20769168 is 20769168 (kafka.consumer.PartitionTopicInfo) [2013-01-11 09:40:47,705] DEBUG initial fetch offset of topic-24:3: fetched offset = 20769168: consumed offset = 20769168 is 20769168 (kafka.consumer.PartitionTopicInfo) [2013-01-11 09:40:47,705] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], topic-24:3: fetched offset = 20769168: consumed offset = 20769168 selected new offset 20769168 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:47,705] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], Consumer KafkaMirror_mirror-01-1357893636873-1b343568 rebalancing the following partitions: ArrayBuffer(0) for topic topic-9 with consumers: List(KafkaMirror_mirror-01-1357893636873-1b343568-0) (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:47,705] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], KafkaMirror_mirror-01-1357893636873-1b343568-0 attempting to claim partition 0 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:47,707] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], exception during rebalance (kafka.consumer.ZookeeperConsumerConnector) kafka.common.NoBrokersForPartitionException: No leader available for partition 0 on topic topic-9 at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.kafka$consumer$ZookeeperConsumerConnector$ZKRebalancerListener$$addPartitionTopicInfo(ZookeeperConsumerConnector.scala:584) at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$kafka$consumer$ZookeeperConsumerConnector$ZKRebalancerListener$$rebalance$2$$anonfun$apply$11$$anonfun$apply$13.apply(ZookeeperConsumerConnector.scala:459) at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$kafka$consumer$ZookeeperConsumerConnector$ZKRebalancerListener$$rebalance$2$$anonfun$apply$11$$anonfun$apply$13.apply(ZookeeperConsumerConnector.scala:456) at scala.collection.immutable.Range$ByOne$class.foreach(Range.scala:285) at scala.collection.immutable.Range$$anon$2.foreach(Range.scala:265) at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$kafka$consumer$ZookeeperConsumerConnector$ZKRebalancerListener$$rebalance$2$$anonfun$apply$11.apply(ZookeeperConsumerConnector.scala:456) at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$kafka$consumer$ZookeeperConsumerConnector$ZKRebalancerListener$$rebalance$2$$anonfun$apply$11.apply(ZookeeperConsumerConnector.scala:443) at scala.collection.mutable.HashSet.foreach(HashSet.scala:61) at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$kafka$consumer$ZookeeperConsumerConnector$ZKRebalancerListener$$rebalance$2.apply(ZookeeperConsumerConnector.scala:443) at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$kafka$consumer$ZookeeperConsumerConnector$ZKRebalancerListener$$rebalance$2.apply(ZookeeperConsumerConnector.scala:430) at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:80) at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:80) at scala.collection.Iterator$class.foreach(Iterator.scala:631) at scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:161) at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:194) at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39) at scala.collection.mutable.HashMap.foreach(HashMap.scala:80) at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.kafka$consumer$ZookeeperConsumerConnector$ZKRebalancerListener$$rebalance(ZookeeperConsumerConnector.scala:430) at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$syncedRebalance$1.apply$mcVI$sp(ZookeeperConsumerConnector.scala:373) at scala.collection.immutable.Range$ByOne$class.foreach$mVc$sp(Range.scala:282) at scala.collection.immutable.Range$$anon$2.foreach$mVc$sp(Range.scala:265) at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:368) at kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$reinitializeConsumer(ZookeeperConsumerConnector.scala:697) at kafka.consumer.ZookeeperConsumerConnector$WildcardStreamsHandler.(ZookeeperConsumerConnector.scala:732) at kafka.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:141) at kafka.tools.MirrorMaker$$anonfun$4.apply(MirrorMaker.scala:116) at kafka.tools.MirrorMaker$$anonfun$4.apply(MirrorMaker.scala:116) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61) at scala.collection.immutable.List.foreach(List.scala:45) at scala.collection.TraversableLike$class.map(TraversableLike.scala:206) at scala.collection.immutable.List.map(List.scala:45) at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:116) at kafka.tools.MirrorMaker.main(MirrorMaker.scala) [2013-01-11 09:40:47,709] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], end rebalancing consumer KafkaMirror_mirror-01-1357893636873-1b343568 try #0 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:47,709] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], Rebalancing attempt failed. Clearing the cache before the next rebalancing operation is triggered (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:47,710] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], Cleared all relevant queues for this fetcher (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:47,710] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:47,710] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:47,710] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:47,711] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:47,711] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:47,711] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:47,711] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:47,711] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:47,711] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:47,711] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:47,711] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:47,712] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:47,712] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:47,712] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:47,712] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:47,712] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:47,712] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:47,712] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:47,712] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:47,713] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:47,713] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:47,713] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:47,713] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:47,713] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:47,713] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], Cleared the data chunks in all the consumer message iterators (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:47,713] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], Committing all offsets after clearing the fetcher queues (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:49,705] DEBUG Got ping response for sessionid: 0x13c012a599601aa after 144ms (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:49,716] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], begin rebalancing consumer KafkaMirror_mirror-01-1357893636873-1b343568 try #1 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:49,860] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 69,8 replyHeader:: 69,420842,0 request:: '/brokers/ids,F response:: v{'174984962} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:50,005] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 70,4 replyHeader:: 70,420842,0 request:: '/brokers/ids/174984962,F response:: #6c6f672d6c3131302d31352d322e63726974656f2e70726f643a393039323a39393939,s{302210,302210,1357665300415,1357665300415,0,0,0,88947374043824448,35,0,302210} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:50,150] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 71,4 replyHeader:: 71,420842,0 request:: '/consumers/KafkaMirror/ids/KafkaMirror_mirror-01-1357893636873-1b343568,F response:: #2131217864756d6d7978,s{420842,420842,1357893637481,1357893637481,0,0,0,88947374043824554,10,0,420842} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:50,151] INFO Constructing topic count for KafkaMirror_mirror-01-1357893636873-1b343568 from !1!xdummyx using !(\p{Digit}+)!(.*) as pattern. (kafka.consumer.TopicCount$) [2013-01-11 09:40:50,296] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 72,8 replyHeader:: 72,420842,0 request:: '/brokers/topics,F response:: v{'topic-23,'topic-5,'topic-1,'topic-7,'topic-14,'topic-6,'topic-8,'topic-92,'topic-9al,'topic-3,'topic-2,'topic-15,'topic-19,'dummy,'topic-24,'topic-20,'topic-16,'topic-18,'topic-21,'topic-3,'topic-22,'topic-25,'topic-17,'topic-9} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:50,296] DEBUG topic-23 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:50,297] DEBUG topic-5 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:50,297] DEBUG topic-1 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:50,297] DEBUG topic-7 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:50,297] DEBUG topic-14 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:50,297] DEBUG topic-6 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:50,298] DEBUG topic-8 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:50,298] DEBUG topic-92 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:50,298] DEBUG topic-9al allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:50,298] DEBUG topic-3 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:50,299] DEBUG topic-2 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:50,299] DEBUG topic-15 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:50,299] DEBUG topic-19 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:50,299] DEBUG dummy allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:50,299] DEBUG topic-24 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:50,300] DEBUG topic-20 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:50,300] DEBUG topic-16 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:50,300] DEBUG topic-18 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:50,300] DEBUG topic-21 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:50,300] DEBUG topic-3 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:50,301] DEBUG topic-22 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:50,301] DEBUG topic-25 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:50,301] DEBUG topic-17 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:50,301] DEBUG topic-9 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:50,447] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 73,8 replyHeader:: 73,420842,0 request:: '/consumers/KafkaMirror/ids,T response:: v{'KafkaMirror_mirror-01-1357893636873-1b343568} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:50,592] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 74,4 replyHeader:: 74,420842,0 request:: '/consumers/KafkaMirror/ids/KafkaMirror_mirror-01-1357893636873-1b343568,F response:: #2131217864756d6d7978,s{420842,420842,1357893637481,1357893637481,0,0,0,88947374043824554,10,0,420842} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:50,592] INFO Constructing topic count for KafkaMirror_mirror-01-1357893636873-1b343568 from !1!xdummyx using !(\p{Digit}+)!(.*) as pattern. (kafka.consumer.TopicCount$) [2013-01-11 09:40:50,738] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 75,8 replyHeader:: 75,420842,0 request:: '/brokers/topics,F response:: v{'topic-23,'topic-5,'topic-1,'topic-7,'topic-14,'topic-6,'topic-8,'topic-92,'topic-9al,'topic-3,'topic-2,'topic-15,'topic-19,'dummy,'topic-24,'topic-20,'topic-16,'topic-18,'topic-21,'topic-3,'topic-22,'topic-25,'topic-17,'topic-9} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:50,738] DEBUG topic-23 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:50,738] DEBUG topic-5 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:50,738] DEBUG topic-1 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:50,739] DEBUG topic-7 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:50,739] DEBUG topic-14 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:50,739] DEBUG topic-6 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:50,739] DEBUG topic-8 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:50,739] DEBUG topic-92 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:50,740] DEBUG topic-9al allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:50,740] DEBUG topic-3 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:50,740] DEBUG topic-2 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:50,740] DEBUG topic-15 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:50,740] DEBUG topic-19 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:50,741] DEBUG dummy allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:50,741] DEBUG topic-24 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:50,741] DEBUG topic-20 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:50,741] DEBUG topic-16 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:50,742] DEBUG topic-18 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:50,742] DEBUG topic-21 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:50,742] DEBUG topic-3 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:50,742] DEBUG topic-22 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:50,742] DEBUG topic-25 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:50,743] DEBUG topic-17 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:50,743] DEBUG topic-9 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:50,889] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 76,8 replyHeader:: 76,420842,0 request:: '/brokers/ids,F response:: v{'174984962} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:51,034] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 77,4 replyHeader:: 77,420842,0 request:: '/brokers/ids/174984962,F response:: #6c6f672d6c3131302d31352d322e63726974656f2e70726f643a393039323a39393939,s{302210,302210,1357665300415,1357665300415,0,0,0,88947374043824448,35,0,302210} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:51,034] INFO Verifying properties (kafka.utils.VerifiableProperties) [2013-01-11 09:40:51,035] INFO Property broker.list is overridden to broker-01:9092 (kafka.utils.VerifiableProperties) [2013-01-11 09:40:51,035] INFO Property clientid is overridden to KafkaMirror (kafka.utils.VerifiableProperties) [2013-01-11 09:40:51,035] INFO Fetching metadata for topic Set(topic-1, topic-8, topic-24, topic-9, topic-20, topic-9al, topic-16, topic-22, topic-15, topic-5, topic-17, dummy, topic-18, topic-3, topic-7, topic-23, topic-14, topic-92, topic-6, topic-25, topic-21, topic-2, topic-3, topic-19) (kafka.client.ClientUtils$) [2013-01-11 09:40:51,180] DEBUG Created socket with SO_TIMEOUT = 1500 (requested 1500), SO_RCVBUF = 43690 (requested -1), SO_SNDBUF = 102400 (requested 102400). (kafka.network.BlockingChannel) [2013-01-11 09:40:51,180] INFO Connected to broker-01:9092 for producing (kafka.producer.SyncProducer) [2013-01-11 09:40:51,436] INFO Disconnecting from broker-01:9092 (kafka.producer.SyncProducer) [2013-01-11 09:40:51,437] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], Cleared all relevant queues for this fetcher (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:51,437] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:51,438] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:51,438] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:51,438] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:51,438] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:51,438] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:51,438] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:51,438] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:51,438] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:51,439] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:51,439] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:51,439] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:51,439] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:51,439] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:51,439] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:51,439] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:51,440] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:51,440] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:51,440] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:51,440] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:51,440] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:51,440] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:51,440] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:51,440] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:51,441] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], Cleared the data chunks in all the consumer message iterators (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:51,441] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], Committing all offsets after clearing the fetcher queues (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:51,441] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], Releasing partition ownership (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:51,441] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], Consumer KafkaMirror_mirror-01-1357893636873-1b343568 rebalancing the following partitions: ArrayBuffer(0) for topic topic-1 with consumers: List(KafkaMirror_mirror-01-1357893636873-1b343568-0) (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:51,441] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], KafkaMirror_mirror-01-1357893636873-1b343568-0 attempting to claim partition 0 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:51,442] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], Leader for partition 0 for topic topic-1 is 174984962 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:51,587] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 78,4 replyHeader:: 78,420842,0 request:: '/consumers/KafkaMirror/offsets/topic-1/0,F response:: #3339393433343839,s{471,420757,1357229396250,1357893156979,8824,0,0,0,8,0,471} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:51,588] DEBUG initial consumer offset of topic-1:0: fetched offset = 39943489: consumed offset = 39943489 is 39943489 (kafka.consumer.PartitionTopicInfo) [2013-01-11 09:40:51,588] DEBUG initial fetch offset of topic-1:0: fetched offset = 39943489: consumed offset = 39943489 is 39943489 (kafka.consumer.PartitionTopicInfo) [2013-01-11 09:40:51,588] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], topic-1:0: fetched offset = 39943489: consumed offset = 39943489 selected new offset 39943489 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:51,588] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], Consumer KafkaMirror_mirror-01-1357893636873-1b343568 rebalancing the following partitions: ArrayBuffer(0) for topic topic-8 with consumers: List(KafkaMirror_mirror-01-1357893636873-1b343568-0) (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:51,588] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], KafkaMirror_mirror-01-1357893636873-1b343568-0 attempting to claim partition 0 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:51,588] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], Leader for partition 0 for topic topic-8 is 174984962 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:51,733] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 79,4 replyHeader:: 79,420842,0 request:: '/consumers/KafkaMirror/offsets/topic-8/0,F response:: #343431393339,s{474,420758,1357229396698,1357893157124,8824,0,0,0,6,0,474} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:51,733] DEBUG initial consumer offset of topic-8:0: fetched offset = 441939: consumed offset = 441939 is 441939 (kafka.consumer.PartitionTopicInfo) [2013-01-11 09:40:51,734] DEBUG initial fetch offset of topic-8:0: fetched offset = 441939: consumed offset = 441939 is 441939 (kafka.consumer.PartitionTopicInfo) [2013-01-11 09:40:51,734] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], topic-8:0: fetched offset = 441939: consumed offset = 441939 selected new offset 441939 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:51,734] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], Consumer KafkaMirror_mirror-01-1357893636873-1b343568 rebalancing the following partitions: ArrayBuffer(0, 1, 2, 3) for topic topic-24 with consumers: List(KafkaMirror_mirror-01-1357893636873-1b343568-0) (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:51,734] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], KafkaMirror_mirror-01-1357893636873-1b343568-0 attempting to claim partition 0 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:51,734] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], Leader for partition 0 for topic topic-24 is 174984962 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:51,861] DEBUG 5002 ms elapsed. Queue time reached. Sending.. (kafka.producer.async.ProducerSendThread) [2013-01-11 09:40:51,862] DEBUG Handling 0 events (kafka.producer.async.ProducerSendThread) [2013-01-11 09:40:51,879] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 80,4 replyHeader:: 80,420842,0 request:: '/consumers/KafkaMirror/offsets/topic-24/0,F response:: #3230393036373137,s{462,420751,1357229394938,1357893156104,8824,0,0,0,8,0,462} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:51,879] DEBUG initial consumer offset of topic-24:0: fetched offset = 20906717: consumed offset = 20906717 is 20906717 (kafka.consumer.PartitionTopicInfo) [2013-01-11 09:40:51,879] DEBUG initial fetch offset of topic-24:0: fetched offset = 20906717: consumed offset = 20906717 is 20906717 (kafka.consumer.PartitionTopicInfo) [2013-01-11 09:40:51,879] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], topic-24:0: fetched offset = 20906717: consumed offset = 20906717 selected new offset 20906717 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:51,880] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], KafkaMirror_mirror-01-1357893636873-1b343568-0 attempting to claim partition 1 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:51,880] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], Leader for partition 1 for topic topic-24 is 174984962 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:52,025] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 81,4 replyHeader:: 81,420842,0 request:: '/consumers/KafkaMirror/offsets/topic-24/1,F response:: #3230373639333336,s{13571,420753,1357301283528,1357893156396,8807,0,0,0,8,0,13571} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:52,025] DEBUG initial consumer offset of topic-24:1: fetched offset = 20769336: consumed offset = 20769336 is 20769336 (kafka.consumer.PartitionTopicInfo) [2013-01-11 09:40:52,025] DEBUG initial fetch offset of topic-24:1: fetched offset = 20769336: consumed offset = 20769336 is 20769336 (kafka.consumer.PartitionTopicInfo) [2013-01-11 09:40:52,025] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], topic-24:1: fetched offset = 20769336: consumed offset = 20769336 selected new offset 20769336 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:52,026] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], KafkaMirror_mirror-01-1357893636873-1b343568-0 attempting to claim partition 2 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:52,026] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], Leader for partition 2 for topic topic-24 is 174984962 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:52,171] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 82,4 replyHeader:: 82,420842,0 request:: '/consumers/KafkaMirror/offsets/topic-24/2,F response:: #3230373637383233,s{13568,420752,1357301283093,1357893156249,8807,0,0,0,8,0,13568} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:52,171] DEBUG initial consumer offset of topic-24:2: fetched offset = 20767823: consumed offset = 20767823 is 20767823 (kafka.consumer.PartitionTopicInfo) [2013-01-11 09:40:52,171] DEBUG initial fetch offset of topic-24:2: fetched offset = 20767823: consumed offset = 20767823 is 20767823 (kafka.consumer.PartitionTopicInfo) [2013-01-11 09:40:52,171] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], topic-24:2: fetched offset = 20767823: consumed offset = 20767823 selected new offset 20767823 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:52,171] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], KafkaMirror_mirror-01-1357893636873-1b343568-0 attempting to claim partition 3 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:52,171] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], Leader for partition 3 for topic topic-24 is 174984962 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:52,316] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 83,4 replyHeader:: 83,420842,0 request:: '/consumers/KafkaMirror/offsets/topic-24/3,F response:: #3230373639313638,s{13574,420754,1357301283963,1357893156541,8807,0,0,0,8,0,13574} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:52,317] DEBUG initial consumer offset of topic-24:3: fetched offset = 20769168: consumed offset = 20769168 is 20769168 (kafka.consumer.PartitionTopicInfo) [2013-01-11 09:40:52,317] DEBUG initial fetch offset of topic-24:3: fetched offset = 20769168: consumed offset = 20769168 is 20769168 (kafka.consumer.PartitionTopicInfo) [2013-01-11 09:40:52,317] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], topic-24:3: fetched offset = 20769168: consumed offset = 20769168 selected new offset 20769168 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:52,317] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], Consumer KafkaMirror_mirror-01-1357893636873-1b343568 rebalancing the following partitions: ArrayBuffer(0) for topic topic-9 with consumers: List(KafkaMirror_mirror-01-1357893636873-1b343568-0) (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:52,317] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], KafkaMirror_mirror-01-1357893636873-1b343568-0 attempting to claim partition 0 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:52,317] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], exception during rebalance (kafka.consumer.ZookeeperConsumerConnector) kafka.common.NoBrokersForPartitionException: No leader available for partition 0 on topic topic-9 at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.kafka$consumer$ZookeeperConsumerConnector$ZKRebalancerListener$$addPartitionTopicInfo(ZookeeperConsumerConnector.scala:584) at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$kafka$consumer$ZookeeperConsumerConnector$ZKRebalancerListener$$rebalance$2$$anonfun$apply$11$$anonfun$apply$13.apply(ZookeeperConsumerConnector.scala:459) at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$kafka$consumer$ZookeeperConsumerConnector$ZKRebalancerListener$$rebalance$2$$anonfun$apply$11$$anonfun$apply$13.apply(ZookeeperConsumerConnector.scala:456) at scala.collection.immutable.Range$ByOne$class.foreach(Range.scala:285) at scala.collection.immutable.Range$$anon$2.foreach(Range.scala:265) at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$kafka$consumer$ZookeeperConsumerConnector$ZKRebalancerListener$$rebalance$2$$anonfun$apply$11.apply(ZookeeperConsumerConnector.scala:456) at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$kafka$consumer$ZookeeperConsumerConnector$ZKRebalancerListener$$rebalance$2$$anonfun$apply$11.apply(ZookeeperConsumerConnector.scala:443) at scala.collection.mutable.HashSet.foreach(HashSet.scala:61) at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$kafka$consumer$ZookeeperConsumerConnector$ZKRebalancerListener$$rebalance$2.apply(ZookeeperConsumerConnector.scala:443) at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$kafka$consumer$ZookeeperConsumerConnector$ZKRebalancerListener$$rebalance$2.apply(ZookeeperConsumerConnector.scala:430) at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:80) at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:80) at scala.collection.Iterator$class.foreach(Iterator.scala:631) at scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:161) at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:194) at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39) at scala.collection.mutable.HashMap.foreach(HashMap.scala:80) at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.kafka$consumer$ZookeeperConsumerConnector$ZKRebalancerListener$$rebalance(ZookeeperConsumerConnector.scala:430) at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$syncedRebalance$1.apply$mcVI$sp(ZookeeperConsumerConnector.scala:373) at scala.collection.immutable.Range$ByOne$class.foreach$mVc$sp(Range.scala:282) at scala.collection.immutable.Range$$anon$2.foreach$mVc$sp(Range.scala:265) at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:368) at kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$reinitializeConsumer(ZookeeperConsumerConnector.scala:697) at kafka.consumer.ZookeeperConsumerConnector$WildcardStreamsHandler.(ZookeeperConsumerConnector.scala:732) at kafka.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:141) at kafka.tools.MirrorMaker$$anonfun$4.apply(MirrorMaker.scala:116) at kafka.tools.MirrorMaker$$anonfun$4.apply(MirrorMaker.scala:116) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61) at scala.collection.immutable.List.foreach(List.scala:45) at scala.collection.TraversableLike$class.map(TraversableLike.scala:206) at scala.collection.immutable.List.map(List.scala:45) at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:116) at kafka.tools.MirrorMaker.main(MirrorMaker.scala) [2013-01-11 09:40:52,318] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], end rebalancing consumer KafkaMirror_mirror-01-1357893636873-1b343568 try #1 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:52,318] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], Rebalancing attempt failed. Clearing the cache before the next rebalancing operation is triggered (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:52,319] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], Cleared all relevant queues for this fetcher (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:52,319] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:52,319] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:52,319] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:52,319] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:52,319] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:52,319] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:52,319] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:52,319] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:52,319] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:52,320] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:52,320] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:52,320] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:52,320] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:52,320] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:52,320] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:52,320] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:52,320] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:52,321] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:52,321] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:52,321] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:52,321] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:52,321] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:52,321] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:52,321] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:52,321] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], Cleared the data chunks in all the consumer message iterators (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:52,321] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], Committing all offsets after clearing the fetcher queues (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:54,317] DEBUG Got ping response for sessionid: 0x13c012a599601aa after 144ms (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:54,323] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], begin rebalancing consumer KafkaMirror_mirror-01-1357893636873-1b343568 try #2 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:54,468] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 84,8 replyHeader:: 84,420842,0 request:: '/brokers/ids,F response:: v{'174984962} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:54,613] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 85,4 replyHeader:: 85,420842,0 request:: '/brokers/ids/174984962,F response:: #6c6f672d6c3131302d31352d322e63726974656f2e70726f643a393039323a39393939,s{302210,302210,1357665300415,1357665300415,0,0,0,88947374043824448,35,0,302210} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:54,758] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 86,4 replyHeader:: 86,420842,0 request:: '/consumers/KafkaMirror/ids/KafkaMirror_mirror-01-1357893636873-1b343568,F response:: #2131217864756d6d7978,s{420842,420842,1357893637481,1357893637481,0,0,0,88947374043824554,10,0,420842} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:54,758] INFO Constructing topic count for KafkaMirror_mirror-01-1357893636873-1b343568 from !1!xdummyx using !(\p{Digit}+)!(.*) as pattern. (kafka.consumer.TopicCount$) [2013-01-11 09:40:54,904] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 87,8 replyHeader:: 87,420842,0 request:: '/brokers/topics,F response:: v{'topic-23,'topic-5,'topic-1,'topic-7,'topic-14,'topic-6,'topic-8,'topic-92,'topic-9al,'topic-3,'topic-2,'topic-15,'topic-19,'dummy,'topic-24,'topic-20,'topic-16,'topic-18,'topic-21,'topic-3,'topic-22,'topic-25,'topic-17,'topic-9} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:54,904] DEBUG topic-23 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:54,904] DEBUG topic-5 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:54,904] DEBUG topic-1 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:54,905] DEBUG topic-7 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:54,905] DEBUG topic-14 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:54,905] DEBUG topic-6 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:54,905] DEBUG topic-8 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:54,905] DEBUG topic-92 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:54,905] DEBUG topic-9al allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:54,906] DEBUG topic-3 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:54,906] DEBUG topic-2 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:54,906] DEBUG topic-15 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:54,906] DEBUG topic-19 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:54,906] DEBUG dummy allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:54,906] DEBUG topic-24 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:54,907] DEBUG topic-20 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:54,907] DEBUG topic-16 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:54,907] DEBUG topic-18 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:54,907] DEBUG topic-21 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:54,907] DEBUG topic-3 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:54,908] DEBUG topic-22 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:54,908] DEBUG topic-25 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:54,908] DEBUG topic-17 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:54,908] DEBUG topic-9 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:55,055] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 88,8 replyHeader:: 88,420842,0 request:: '/consumers/KafkaMirror/ids,T response:: v{'KafkaMirror_mirror-01-1357893636873-1b343568} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:55,200] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 89,4 replyHeader:: 89,420842,0 request:: '/consumers/KafkaMirror/ids/KafkaMirror_mirror-01-1357893636873-1b343568,F response:: #2131217864756d6d7978,s{420842,420842,1357893637481,1357893637481,0,0,0,88947374043824554,10,0,420842} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:55,200] INFO Constructing topic count for KafkaMirror_mirror-01-1357893636873-1b343568 from !1!xdummyx using !(\p{Digit}+)!(.*) as pattern. (kafka.consumer.TopicCount$) [2013-01-11 09:40:55,346] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 90,8 replyHeader:: 90,420842,0 request:: '/brokers/topics,F response:: v{'topic-23,'topic-5,'topic-1,'topic-7,'topic-14,'topic-6,'topic-8,'topic-92,'topic-9al,'topic-3,'topic-2,'topic-15,'topic-19,'dummy,'topic-24,'topic-20,'topic-16,'topic-18,'topic-21,'topic-3,'topic-22,'topic-25,'topic-17,'topic-9} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:55,346] DEBUG topic-23 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:55,346] DEBUG topic-5 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:55,346] DEBUG topic-1 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:55,347] DEBUG topic-7 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:55,347] DEBUG topic-14 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:55,347] DEBUG topic-6 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:55,347] DEBUG topic-8 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:55,347] DEBUG topic-92 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:55,347] DEBUG topic-9al allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:55,347] DEBUG topic-3 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:55,347] DEBUG topic-2 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:55,348] DEBUG topic-15 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:55,348] DEBUG topic-19 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:55,348] DEBUG dummy allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:55,348] DEBUG topic-24 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:55,348] DEBUG topic-20 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:55,348] DEBUG topic-16 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:55,349] DEBUG topic-18 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:55,349] DEBUG topic-21 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:55,349] DEBUG topic-3 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:55,349] DEBUG topic-22 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:55,349] DEBUG topic-25 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:55,349] DEBUG topic-17 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:55,349] DEBUG topic-9 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:55,496] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 91,8 replyHeader:: 91,420842,0 request:: '/brokers/ids,F response:: v{'174984962} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:55,641] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 92,4 replyHeader:: 92,420842,0 request:: '/brokers/ids/174984962,F response:: #6c6f672d6c3131302d31352d322e63726974656f2e70726f643a393039323a39393939,s{302210,302210,1357665300415,1357665300415,0,0,0,88947374043824448,35,0,302210} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:55,641] INFO Verifying properties (kafka.utils.VerifiableProperties) [2013-01-11 09:40:55,641] INFO Property broker.list is overridden to broker-01:9092 (kafka.utils.VerifiableProperties) [2013-01-11 09:40:55,641] INFO Property clientid is overridden to KafkaMirror (kafka.utils.VerifiableProperties) [2013-01-11 09:40:55,642] INFO Fetching metadata for topic Set(topic-1, topic-8, topic-24, topic-9, topic-20, topic-9al, topic-16, topic-22, topic-15, topic-5, topic-17, dummy, topic-18, topic-3, topic-7, topic-23, topic-14, topic-92, topic-6, topic-25, topic-21, topic-2, topic-3, topic-19) (kafka.client.ClientUtils$) [2013-01-11 09:40:55,787] DEBUG Created socket with SO_TIMEOUT = 1500 (requested 1500), SO_RCVBUF = 43690 (requested -1), SO_SNDBUF = 102400 (requested 102400). (kafka.network.BlockingChannel) [2013-01-11 09:40:55,787] INFO Connected to broker-01:9092 for producing (kafka.producer.SyncProducer) [2013-01-11 09:40:56,049] INFO Disconnecting from broker-01:9092 (kafka.producer.SyncProducer) [2013-01-11 09:40:56,050] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], Cleared all relevant queues for this fetcher (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:56,050] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:56,051] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:56,051] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:56,051] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:56,051] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:56,051] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:56,051] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:56,051] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:56,051] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:56,051] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:56,051] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:56,052] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:56,052] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:56,052] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:56,052] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:56,052] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:56,052] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:56,052] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:56,052] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:56,052] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:56,052] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:56,053] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:56,053] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:56,053] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:56,053] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], Cleared the data chunks in all the consumer message iterators (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:56,053] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], Committing all offsets after clearing the fetcher queues (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:56,053] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], Releasing partition ownership (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:56,053] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], Consumer KafkaMirror_mirror-01-1357893636873-1b343568 rebalancing the following partitions: ArrayBuffer(0) for topic topic-1 with consumers: List(KafkaMirror_mirror-01-1357893636873-1b343568-0) (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:56,053] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], KafkaMirror_mirror-01-1357893636873-1b343568-0 attempting to claim partition 0 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:56,053] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], Leader for partition 0 for topic topic-1 is 174984962 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:56,198] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 93,4 replyHeader:: 93,420842,0 request:: '/consumers/KafkaMirror/offsets/topic-1/0,F response:: #3339393433343839,s{471,420757,1357229396250,1357893156979,8824,0,0,0,8,0,471} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:56,199] DEBUG initial consumer offset of topic-1:0: fetched offset = 39943489: consumed offset = 39943489 is 39943489 (kafka.consumer.PartitionTopicInfo) [2013-01-11 09:40:56,199] DEBUG initial fetch offset of topic-1:0: fetched offset = 39943489: consumed offset = 39943489 is 39943489 (kafka.consumer.PartitionTopicInfo) [2013-01-11 09:40:56,199] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], topic-1:0: fetched offset = 39943489: consumed offset = 39943489 selected new offset 39943489 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:56,199] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], Consumer KafkaMirror_mirror-01-1357893636873-1b343568 rebalancing the following partitions: ArrayBuffer(0) for topic topic-8 with consumers: List(KafkaMirror_mirror-01-1357893636873-1b343568-0) (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:56,199] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], KafkaMirror_mirror-01-1357893636873-1b343568-0 attempting to claim partition 0 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:56,200] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], Leader for partition 0 for topic topic-8 is 174984962 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:56,345] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 94,4 replyHeader:: 94,420842,0 request:: '/consumers/KafkaMirror/offsets/topic-8/0,F response:: #343431393339,s{474,420758,1357229396698,1357893157124,8824,0,0,0,6,0,474} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:56,345] DEBUG initial consumer offset of topic-8:0: fetched offset = 441939: consumed offset = 441939 is 441939 (kafka.consumer.PartitionTopicInfo) [2013-01-11 09:40:56,345] DEBUG initial fetch offset of topic-8:0: fetched offset = 441939: consumed offset = 441939 is 441939 (kafka.consumer.PartitionTopicInfo) [2013-01-11 09:40:56,345] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], topic-8:0: fetched offset = 441939: consumed offset = 441939 selected new offset 441939 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:56,345] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], Consumer KafkaMirror_mirror-01-1357893636873-1b343568 rebalancing the following partitions: ArrayBuffer(0, 1, 2, 3) for topic topic-24 with consumers: List(KafkaMirror_mirror-01-1357893636873-1b343568-0) (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:56,345] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], KafkaMirror_mirror-01-1357893636873-1b343568-0 attempting to claim partition 0 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:56,346] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], Leader for partition 0 for topic topic-24 is 174984962 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:56,491] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 95,4 replyHeader:: 95,420842,0 request:: '/consumers/KafkaMirror/offsets/topic-24/0,F response:: #3230393036373137,s{462,420751,1357229394938,1357893156104,8824,0,0,0,8,0,462} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:56,491] DEBUG initial consumer offset of topic-24:0: fetched offset = 20906717: consumed offset = 20906717 is 20906717 (kafka.consumer.PartitionTopicInfo) [2013-01-11 09:40:56,491] DEBUG initial fetch offset of topic-24:0: fetched offset = 20906717: consumed offset = 20906717 is 20906717 (kafka.consumer.PartitionTopicInfo) [2013-01-11 09:40:56,491] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], topic-24:0: fetched offset = 20906717: consumed offset = 20906717 selected new offset 20906717 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:56,491] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], KafkaMirror_mirror-01-1357893636873-1b343568-0 attempting to claim partition 1 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:56,491] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], Leader for partition 1 for topic topic-24 is 174984962 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:56,636] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 96,4 replyHeader:: 96,420842,0 request:: '/consumers/KafkaMirror/offsets/topic-24/1,F response:: #3230373639333336,s{13571,420753,1357301283528,1357893156396,8807,0,0,0,8,0,13571} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:56,637] DEBUG initial consumer offset of topic-24:1: fetched offset = 20769336: consumed offset = 20769336 is 20769336 (kafka.consumer.PartitionTopicInfo) [2013-01-11 09:40:56,637] DEBUG initial fetch offset of topic-24:1: fetched offset = 20769336: consumed offset = 20769336 is 20769336 (kafka.consumer.PartitionTopicInfo) [2013-01-11 09:40:56,637] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], topic-24:1: fetched offset = 20769336: consumed offset = 20769336 selected new offset 20769336 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:56,637] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], KafkaMirror_mirror-01-1357893636873-1b343568-0 attempting to claim partition 2 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:56,637] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], Leader for partition 2 for topic topic-24 is 174984962 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:56,782] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 97,4 replyHeader:: 97,420842,0 request:: '/consumers/KafkaMirror/offsets/topic-24/2,F response:: #3230373637383233,s{13568,420752,1357301283093,1357893156249,8807,0,0,0,8,0,13568} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:56,782] DEBUG initial consumer offset of topic-24:2: fetched offset = 20767823: consumed offset = 20767823 is 20767823 (kafka.consumer.PartitionTopicInfo) [2013-01-11 09:40:56,782] DEBUG initial fetch offset of topic-24:2: fetched offset = 20767823: consumed offset = 20767823 is 20767823 (kafka.consumer.PartitionTopicInfo) [2013-01-11 09:40:56,782] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], topic-24:2: fetched offset = 20767823: consumed offset = 20767823 selected new offset 20767823 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:56,783] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], KafkaMirror_mirror-01-1357893636873-1b343568-0 attempting to claim partition 3 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:56,783] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], Leader for partition 3 for topic topic-24 is 174984962 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:56,863] DEBUG 5001 ms elapsed. Queue time reached. Sending.. (kafka.producer.async.ProducerSendThread) [2013-01-11 09:40:56,863] DEBUG Handling 0 events (kafka.producer.async.ProducerSendThread) [2013-01-11 09:40:56,928] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 98,4 replyHeader:: 98,420842,0 request:: '/consumers/KafkaMirror/offsets/topic-24/3,F response:: #3230373639313638,s{13574,420754,1357301283963,1357893156541,8807,0,0,0,8,0,13574} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:56,928] DEBUG initial consumer offset of topic-24:3: fetched offset = 20769168: consumed offset = 20769168 is 20769168 (kafka.consumer.PartitionTopicInfo) [2013-01-11 09:40:56,928] DEBUG initial fetch offset of topic-24:3: fetched offset = 20769168: consumed offset = 20769168 is 20769168 (kafka.consumer.PartitionTopicInfo) [2013-01-11 09:40:56,928] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], topic-24:3: fetched offset = 20769168: consumed offset = 20769168 selected new offset 20769168 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:56,928] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], Consumer KafkaMirror_mirror-01-1357893636873-1b343568 rebalancing the following partitions: ArrayBuffer(0) for topic topic-9 with consumers: List(KafkaMirror_mirror-01-1357893636873-1b343568-0) (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:56,928] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], KafkaMirror_mirror-01-1357893636873-1b343568-0 attempting to claim partition 0 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:56,929] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], exception during rebalance (kafka.consumer.ZookeeperConsumerConnector) kafka.common.NoBrokersForPartitionException: No leader available for partition 0 on topic topic-9 at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.kafka$consumer$ZookeeperConsumerConnector$ZKRebalancerListener$$addPartitionTopicInfo(ZookeeperConsumerConnector.scala:584) at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$kafka$consumer$ZookeeperConsumerConnector$ZKRebalancerListener$$rebalance$2$$anonfun$apply$11$$anonfun$apply$13.apply(ZookeeperConsumerConnector.scala:459) at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$kafka$consumer$ZookeeperConsumerConnector$ZKRebalancerListener$$rebalance$2$$anonfun$apply$11$$anonfun$apply$13.apply(ZookeeperConsumerConnector.scala:456) at scala.collection.immutable.Range$ByOne$class.foreach(Range.scala:285) at scala.collection.immutable.Range$$anon$2.foreach(Range.scala:265) at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$kafka$consumer$ZookeeperConsumerConnector$ZKRebalancerListener$$rebalance$2$$anonfun$apply$11.apply(ZookeeperConsumerConnector.scala:456) at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$kafka$consumer$ZookeeperConsumerConnector$ZKRebalancerListener$$rebalance$2$$anonfun$apply$11.apply(ZookeeperConsumerConnector.scala:443) at scala.collection.mutable.HashSet.foreach(HashSet.scala:61) at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$kafka$consumer$ZookeeperConsumerConnector$ZKRebalancerListener$$rebalance$2.apply(ZookeeperConsumerConnector.scala:443) at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$kafka$consumer$ZookeeperConsumerConnector$ZKRebalancerListener$$rebalance$2.apply(ZookeeperConsumerConnector.scala:430) at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:80) at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:80) at scala.collection.Iterator$class.foreach(Iterator.scala:631) at scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:161) at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:194) at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39) at scala.collection.mutable.HashMap.foreach(HashMap.scala:80) at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.kafka$consumer$ZookeeperConsumerConnector$ZKRebalancerListener$$rebalance(ZookeeperConsumerConnector.scala:430) at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$syncedRebalance$1.apply$mcVI$sp(ZookeeperConsumerConnector.scala:373) at scala.collection.immutable.Range$ByOne$class.foreach$mVc$sp(Range.scala:282) at scala.collection.immutable.Range$$anon$2.foreach$mVc$sp(Range.scala:265) at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:368) at kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$reinitializeConsumer(ZookeeperConsumerConnector.scala:697) at kafka.consumer.ZookeeperConsumerConnector$WildcardStreamsHandler.(ZookeeperConsumerConnector.scala:732) at kafka.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:141) at kafka.tools.MirrorMaker$$anonfun$4.apply(MirrorMaker.scala:116) at kafka.tools.MirrorMaker$$anonfun$4.apply(MirrorMaker.scala:116) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61) at scala.collection.immutable.List.foreach(List.scala:45) at scala.collection.TraversableLike$class.map(TraversableLike.scala:206) at scala.collection.immutable.List.map(List.scala:45) at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:116) at kafka.tools.MirrorMaker.main(MirrorMaker.scala) [2013-01-11 09:40:56,929] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], end rebalancing consumer KafkaMirror_mirror-01-1357893636873-1b343568 try #2 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:56,929] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], Rebalancing attempt failed. Clearing the cache before the next rebalancing operation is triggered (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:56,930] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], Cleared all relevant queues for this fetcher (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:56,930] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:56,930] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:56,930] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:56,930] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:56,930] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:56,930] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:56,930] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:56,930] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:56,931] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:56,931] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:56,931] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:56,931] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:56,931] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:56,931] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:56,931] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:56,931] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:56,931] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:56,931] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:56,931] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:56,932] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:56,932] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:56,932] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:56,932] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:56,932] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:40:56,932] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], Cleared the data chunks in all the consumer message iterators (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:56,932] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], Committing all offsets after clearing the fetcher queues (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:58,928] DEBUG Got ping response for sessionid: 0x13c012a599601aa after 144ms (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:58,934] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], begin rebalancing consumer KafkaMirror_mirror-01-1357893636873-1b343568 try #3 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:40:59,079] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 99,8 replyHeader:: 99,420842,0 request:: '/brokers/ids,F response:: v{'174984962} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:59,224] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 100,4 replyHeader:: 100,420842,0 request:: '/brokers/ids/174984962,F response:: #6c6f672d6c3131302d31352d322e63726974656f2e70726f643a393039323a39393939,s{302210,302210,1357665300415,1357665300415,0,0,0,88947374043824448,35,0,302210} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:59,369] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 101,4 replyHeader:: 101,420842,0 request:: '/consumers/KafkaMirror/ids/KafkaMirror_mirror-01-1357893636873-1b343568,F response:: #2131217864756d6d7978,s{420842,420842,1357893637481,1357893637481,0,0,0,88947374043824554,10,0,420842} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:59,369] INFO Constructing topic count for KafkaMirror_mirror-01-1357893636873-1b343568 from !1!xdummyx using !(\p{Digit}+)!(.*) as pattern. (kafka.consumer.TopicCount$) [2013-01-11 09:40:59,515] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 102,8 replyHeader:: 102,420842,0 request:: '/brokers/topics,F response:: v{'topic-23,'topic-5,'topic-1,'topic-7,'topic-14,'topic-6,'topic-8,'topic-92,'topic-9al,'topic-3,'topic-2,'topic-15,'topic-19,'dummy,'topic-24,'topic-20,'topic-16,'topic-18,'topic-21,'topic-3,'topic-22,'topic-25,'topic-17,'topic-9} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:59,515] DEBUG topic-23 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:59,515] DEBUG topic-5 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:59,516] DEBUG topic-1 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:59,516] DEBUG topic-7 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:59,516] DEBUG topic-14 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:59,516] DEBUG topic-6 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:59,516] DEBUG topic-8 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:59,516] DEBUG topic-92 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:59,517] DEBUG topic-9al allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:59,517] DEBUG topic-3 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:59,517] DEBUG topic-2 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:59,517] DEBUG topic-15 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:59,517] DEBUG topic-19 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:59,517] DEBUG dummy allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:59,517] DEBUG topic-24 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:59,518] DEBUG topic-20 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:59,518] DEBUG topic-16 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:59,518] DEBUG topic-18 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:59,518] DEBUG topic-21 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:59,518] DEBUG topic-3 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:59,518] DEBUG topic-22 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:59,519] DEBUG topic-25 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:59,519] DEBUG topic-17 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:59,519] DEBUG topic-9 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:59,665] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 103,8 replyHeader:: 103,420842,0 request:: '/consumers/KafkaMirror/ids,T response:: v{'KafkaMirror_mirror-01-1357893636873-1b343568} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:59,810] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 104,4 replyHeader:: 104,420842,0 request:: '/consumers/KafkaMirror/ids/KafkaMirror_mirror-01-1357893636873-1b343568,F response:: #2131217864756d6d7978,s{420842,420842,1357893637481,1357893637481,0,0,0,88947374043824554,10,0,420842} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:59,810] INFO Constructing topic count for KafkaMirror_mirror-01-1357893636873-1b343568 from !1!xdummyx using !(\p{Digit}+)!(.*) as pattern. (kafka.consumer.TopicCount$) [2013-01-11 09:40:59,956] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 105,8 replyHeader:: 105,420842,0 request:: '/brokers/topics,F response:: v{'topic-23,'topic-5,'topic-1,'topic-7,'topic-14,'topic-6,'topic-8,'topic-92,'topic-9al,'topic-3,'topic-2,'topic-15,'topic-19,'dummy,'topic-24,'topic-20,'topic-16,'topic-18,'topic-21,'topic-3,'topic-22,'topic-25,'topic-17,'topic-9} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:40:59,956] DEBUG topic-23 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:59,956] DEBUG topic-5 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:59,956] DEBUG topic-1 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:59,957] DEBUG topic-7 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:59,957] DEBUG topic-14 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:59,957] DEBUG topic-6 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:59,957] DEBUG topic-8 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:59,957] DEBUG topic-92 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:59,957] DEBUG topic-9al allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:59,957] DEBUG topic-3 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:59,958] DEBUG topic-2 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:59,958] DEBUG topic-15 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:59,958] DEBUG topic-19 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:59,958] DEBUG dummy allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:59,958] DEBUG topic-24 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:59,958] DEBUG topic-20 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:59,959] DEBUG topic-16 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:59,959] DEBUG topic-18 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:59,959] DEBUG topic-21 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:59,959] DEBUG topic-3 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:59,959] DEBUG topic-22 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:59,959] DEBUG topic-25 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:59,959] DEBUG topic-17 allowed (kafka.consumer.Blacklist) [2013-01-11 09:40:59,960] DEBUG topic-9 allowed (kafka.consumer.Blacklist) [2013-01-11 09:41:00,106] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 106,8 replyHeader:: 106,420842,0 request:: '/brokers/ids,F response:: v{'174984962} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:41:00,251] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 107,4 replyHeader:: 107,420842,0 request:: '/brokers/ids/174984962,F response:: #6c6f672d6c3131302d31352d322e63726974656f2e70726f643a393039323a39393939,s{302210,302210,1357665300415,1357665300415,0,0,0,88947374043824448,35,0,302210} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:41:00,251] INFO Verifying properties (kafka.utils.VerifiableProperties) [2013-01-11 09:41:00,251] INFO Property broker.list is overridden to broker-01:9092 (kafka.utils.VerifiableProperties) [2013-01-11 09:41:00,252] INFO Property clientid is overridden to KafkaMirror (kafka.utils.VerifiableProperties) [2013-01-11 09:41:00,252] INFO Fetching metadata for topic Set(topic-1, topic-8, topic-24, topic-9, topic-20, topic-9al, topic-16, topic-22, topic-15, topic-5, topic-17, dummy, topic-18, topic-3, topic-7, topic-23, topic-14, topic-92, topic-6, topic-25, topic-21, topic-2, topic-3, topic-19) (kafka.client.ClientUtils$) [2013-01-11 09:41:00,397] DEBUG Created socket with SO_TIMEOUT = 1500 (requested 1500), SO_RCVBUF = 43690 (requested -1), SO_SNDBUF = 102400 (requested 102400). (kafka.network.BlockingChannel) [2013-01-11 09:41:00,397] INFO Connected to broker-01:9092 for producing (kafka.producer.SyncProducer) [2013-01-11 09:41:00,641] INFO Disconnecting from broker-01:9092 (kafka.producer.SyncProducer) [2013-01-11 09:41:00,643] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], Cleared all relevant queues for this fetcher (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:41:00,643] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:41:00,643] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:41:00,643] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:41:00,643] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:41:00,643] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:41:00,643] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:41:00,643] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:41:00,643] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:41:00,643] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:41:00,643] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:41:00,643] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:41:00,643] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:41:00,644] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:41:00,644] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:41:00,644] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:41:00,644] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:41:00,644] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:41:00,644] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:41:00,644] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:41:00,644] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:41:00,644] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:41:00,644] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:41:00,644] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:41:00,644] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:41:00,644] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], Cleared the data chunks in all the consumer message iterators (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:41:00,644] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], Committing all offsets after clearing the fetcher queues (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:41:00,645] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], Releasing partition ownership (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:41:00,645] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], Consumer KafkaMirror_mirror-01-1357893636873-1b343568 rebalancing the following partitions: ArrayBuffer(0) for topic topic-1 with consumers: List(KafkaMirror_mirror-01-1357893636873-1b343568-0) (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:41:00,645] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], KafkaMirror_mirror-01-1357893636873-1b343568-0 attempting to claim partition 0 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:41:00,645] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], Leader for partition 0 for topic topic-1 is 174984962 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:41:00,790] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 108,4 replyHeader:: 108,420842,0 request:: '/consumers/KafkaMirror/offsets/topic-1/0,F response:: #3339393433343839,s{471,420757,1357229396250,1357893156979,8824,0,0,0,8,0,471} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:41:00,790] DEBUG initial consumer offset of topic-1:0: fetched offset = 39943489: consumed offset = 39943489 is 39943489 (kafka.consumer.PartitionTopicInfo) [2013-01-11 09:41:00,790] DEBUG initial fetch offset of topic-1:0: fetched offset = 39943489: consumed offset = 39943489 is 39943489 (kafka.consumer.PartitionTopicInfo) [2013-01-11 09:41:00,790] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], topic-1:0: fetched offset = 39943489: consumed offset = 39943489 selected new offset 39943489 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:41:00,791] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], Consumer KafkaMirror_mirror-01-1357893636873-1b343568 rebalancing the following partitions: ArrayBuffer(0) for topic topic-8 with consumers: List(KafkaMirror_mirror-01-1357893636873-1b343568-0) (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:41:00,791] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], KafkaMirror_mirror-01-1357893636873-1b343568-0 attempting to claim partition 0 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:41:00,791] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], Leader for partition 0 for topic topic-8 is 174984962 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:41:00,936] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 109,4 replyHeader:: 109,420842,0 request:: '/consumers/KafkaMirror/offsets/topic-8/0,F response:: #343431393339,s{474,420758,1357229396698,1357893157124,8824,0,0,0,6,0,474} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:41:00,936] DEBUG initial consumer offset of topic-8:0: fetched offset = 441939: consumed offset = 441939 is 441939 (kafka.consumer.PartitionTopicInfo) [2013-01-11 09:41:00,936] DEBUG initial fetch offset of topic-8:0: fetched offset = 441939: consumed offset = 441939 is 441939 (kafka.consumer.PartitionTopicInfo) [2013-01-11 09:41:00,936] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], topic-8:0: fetched offset = 441939: consumed offset = 441939 selected new offset 441939 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:41:00,936] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], Consumer KafkaMirror_mirror-01-1357893636873-1b343568 rebalancing the following partitions: ArrayBuffer(0, 1, 2, 3) for topic topic-24 with consumers: List(KafkaMirror_mirror-01-1357893636873-1b343568-0) (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:41:00,936] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], KafkaMirror_mirror-01-1357893636873-1b343568-0 attempting to claim partition 0 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:41:00,937] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], Leader for partition 0 for topic topic-24 is 174984962 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:41:01,081] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 110,4 replyHeader:: 110,420842,0 request:: '/consumers/KafkaMirror/offsets/topic-24/0,F response:: #3230393036373137,s{462,420751,1357229394938,1357893156104,8824,0,0,0,8,0,462} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:41:01,082] DEBUG initial consumer offset of topic-24:0: fetched offset = 20906717: consumed offset = 20906717 is 20906717 (kafka.consumer.PartitionTopicInfo) [2013-01-11 09:41:01,082] DEBUG initial fetch offset of topic-24:0: fetched offset = 20906717: consumed offset = 20906717 is 20906717 (kafka.consumer.PartitionTopicInfo) [2013-01-11 09:41:01,082] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], topic-24:0: fetched offset = 20906717: consumed offset = 20906717 selected new offset 20906717 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:41:01,082] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], KafkaMirror_mirror-01-1357893636873-1b343568-0 attempting to claim partition 1 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:41:01,082] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], Leader for partition 1 for topic topic-24 is 174984962 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:41:01,227] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 111,4 replyHeader:: 111,420842,0 request:: '/consumers/KafkaMirror/offsets/topic-24/1,F response:: #3230373639333336,s{13571,420753,1357301283528,1357893156396,8807,0,0,0,8,0,13571} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:41:01,227] DEBUG initial consumer offset of topic-24:1: fetched offset = 20769336: consumed offset = 20769336 is 20769336 (kafka.consumer.PartitionTopicInfo) [2013-01-11 09:41:01,228] DEBUG initial fetch offset of topic-24:1: fetched offset = 20769336: consumed offset = 20769336 is 20769336 (kafka.consumer.PartitionTopicInfo) [2013-01-11 09:41:01,228] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], topic-24:1: fetched offset = 20769336: consumed offset = 20769336 selected new offset 20769336 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:41:01,228] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], KafkaMirror_mirror-01-1357893636873-1b343568-0 attempting to claim partition 2 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:41:01,228] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], Leader for partition 2 for topic topic-24 is 174984962 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:41:01,373] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 112,4 replyHeader:: 112,420842,0 request:: '/consumers/KafkaMirror/offsets/topic-24/2,F response:: #3230373637383233,s{13568,420752,1357301283093,1357893156249,8807,0,0,0,8,0,13568} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:41:01,373] DEBUG initial consumer offset of topic-24:2: fetched offset = 20767823: consumed offset = 20767823 is 20767823 (kafka.consumer.PartitionTopicInfo) [2013-01-11 09:41:01,373] DEBUG initial fetch offset of topic-24:2: fetched offset = 20767823: consumed offset = 20767823 is 20767823 (kafka.consumer.PartitionTopicInfo) [2013-01-11 09:41:01,373] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], topic-24:2: fetched offset = 20767823: consumed offset = 20767823 selected new offset 20767823 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:41:01,373] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], KafkaMirror_mirror-01-1357893636873-1b343568-0 attempting to claim partition 3 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:41:01,374] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], Leader for partition 3 for topic topic-24 is 174984962 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:41:01,519] DEBUG Reading reply sessionid:0x13c012a599601aa, packet:: clientPath:null serverPath:null finished:false header:: 113,4 replyHeader:: 113,420842,0 request:: '/consumers/KafkaMirror/offsets/topic-24/3,F response:: #3230373639313638,s{13574,420754,1357301283963,1357893156541,8807,0,0,0,8,0,13574} (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:41:01,519] DEBUG initial consumer offset of topic-24:3: fetched offset = 20769168: consumed offset = 20769168 is 20769168 (kafka.consumer.PartitionTopicInfo) [2013-01-11 09:41:01,519] DEBUG initial fetch offset of topic-24:3: fetched offset = 20769168: consumed offset = 20769168 is 20769168 (kafka.consumer.PartitionTopicInfo) [2013-01-11 09:41:01,519] DEBUG [KafkaMirror_mirror-01-1357893636873-1b343568], topic-24:3: fetched offset = 20769168: consumed offset = 20769168 selected new offset 20769168 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:41:01,519] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], Consumer KafkaMirror_mirror-01-1357893636873-1b343568 rebalancing the following partitions: ArrayBuffer(0) for topic topic-9 with consumers: List(KafkaMirror_mirror-01-1357893636873-1b343568-0) (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:41:01,519] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], KafkaMirror_mirror-01-1357893636873-1b343568-0 attempting to claim partition 0 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:41:01,519] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], exception during rebalance (kafka.consumer.ZookeeperConsumerConnector) kafka.common.NoBrokersForPartitionException: No leader available for partition 0 on topic topic-9 at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.kafka$consumer$ZookeeperConsumerConnector$ZKRebalancerListener$$addPartitionTopicInfo(ZookeeperConsumerConnector.scala:584) at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$kafka$consumer$ZookeeperConsumerConnector$ZKRebalancerListener$$rebalance$2$$anonfun$apply$11$$anonfun$apply$13.apply(ZookeeperConsumerConnector.scala:459) at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$kafka$consumer$ZookeeperConsumerConnector$ZKRebalancerListener$$rebalance$2$$anonfun$apply$11$$anonfun$apply$13.apply(ZookeeperConsumerConnector.scala:456) at scala.collection.immutable.Range$ByOne$class.foreach(Range.scala:285) at scala.collection.immutable.Range$$anon$2.foreach(Range.scala:265) at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$kafka$consumer$ZookeeperConsumerConnector$ZKRebalancerListener$$rebalance$2$$anonfun$apply$11.apply(ZookeeperConsumerConnector.scala:456) at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$kafka$consumer$ZookeeperConsumerConnector$ZKRebalancerListener$$rebalance$2$$anonfun$apply$11.apply(ZookeeperConsumerConnector.scala:443) at scala.collection.mutable.HashSet.foreach(HashSet.scala:61) at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$kafka$consumer$ZookeeperConsumerConnector$ZKRebalancerListener$$rebalance$2.apply(ZookeeperConsumerConnector.scala:443) at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$kafka$consumer$ZookeeperConsumerConnector$ZKRebalancerListener$$rebalance$2.apply(ZookeeperConsumerConnector.scala:430) at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:80) at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:80) at scala.collection.Iterator$class.foreach(Iterator.scala:631) at scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:161) at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:194) at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39) at scala.collection.mutable.HashMap.foreach(HashMap.scala:80) at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.kafka$consumer$ZookeeperConsumerConnector$ZKRebalancerListener$$rebalance(ZookeeperConsumerConnector.scala:430) at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$syncedRebalance$1.apply$mcVI$sp(ZookeeperConsumerConnector.scala:373) at scala.collection.immutable.Range$ByOne$class.foreach$mVc$sp(Range.scala:285) at scala.collection.immutable.Range$$anon$2.foreach$mVc$sp(Range.scala:265) at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:368) at kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$reinitializeConsumer(ZookeeperConsumerConnector.scala:697) at kafka.consumer.ZookeeperConsumerConnector$WildcardStreamsHandler.(ZookeeperConsumerConnector.scala:732) at kafka.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:141) at kafka.tools.MirrorMaker$$anonfun$4.apply(MirrorMaker.scala:116) at kafka.tools.MirrorMaker$$anonfun$4.apply(MirrorMaker.scala:116) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61) at scala.collection.immutable.List.foreach(List.scala:45) at scala.collection.TraversableLike$class.map(TraversableLike.scala:206) at scala.collection.immutable.List.map(List.scala:45) at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:116) at kafka.tools.MirrorMaker.main(MirrorMaker.scala) [2013-01-11 09:41:01,520] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], end rebalancing consumer KafkaMirror_mirror-01-1357893636873-1b343568 try #3 (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:41:01,520] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], Rebalancing attempt failed. Clearing the cache before the next rebalancing operation is triggered (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:41:01,520] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], Cleared all relevant queues for this fetcher (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:41:01,520] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:41:01,521] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:41:01,521] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:41:01,521] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:41:01,521] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:41:01,521] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:41:01,521] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:41:01,521] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:41:01,521] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:41:01,521] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:41:01,521] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:41:01,521] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:41:01,521] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:41:01,521] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:41:01,522] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:41:01,522] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:41:01,522] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:41:01,522] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:41:01,522] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:41:01,522] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:41:01,522] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:41:01,522] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:41:01,522] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:41:01,522] INFO Clearing the current data chunk for this consumer iterator (kafka.consumer.ConsumerIterator) [2013-01-11 09:41:01,522] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], Cleared the data chunks in all the consumer message iterators (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:41:01,522] INFO [KafkaMirror_mirror-01-1357893636873-1b343568], Committing all offsets after clearing the fetcher queues (kafka.consumer.ZookeeperConsumerConnector) [2013-01-11 09:41:01,866] DEBUG 5003 ms elapsed. Queue time reached. Sending.. (kafka.producer.async.ProducerSendThread) [2013-01-11 09:41:01,866] DEBUG Handling 0 events (kafka.producer.async.ProducerSendThread) [2013-01-11 09:41:03,519] DEBUG Got ping response for sessionid: 0x13c012a599601aa after 144ms (org.apache.zookeeper.ClientCnxn) Exception in thread "main" kafka.common.ConsumerRebalanceFailedException: KafkaMirror_mirror-01-1357893636873-1b343568 can't rebalance after 4 retries at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:396) at kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$reinitializeConsumer(ZookeeperConsumerConnector.scala:697) at kafka.consumer.ZookeeperConsumerConnector$WildcardStreamsHandler.(ZookeeperConsumerConnector.scala:732) at kafka.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:141) at kafka.tools.MirrorMaker$$anonfun$4.apply(MirrorMaker.scala:116) at kafka.tools.MirrorMaker$$anonfun$4.apply(MirrorMaker.scala:116) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61) at scala.collection.immutable.List.foreach(List.scala:45) at scala.collection.TraversableLike$class.map(TraversableLike.scala:206) at scala.collection.immutable.List.map(List.scala:45) at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:116) at kafka.tools.MirrorMaker.main(MirrorMaker.scala) [2013-01-11 09:41:05,520] DEBUG Got ping response for sessionid: 0x13c012a599601aa after 144ms (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:41:06,868] DEBUG 5002 ms elapsed. Queue time reached. Sending.. (kafka.producer.async.ProducerSendThread) [2013-01-11 09:41:06,869] DEBUG Handling 0 events (kafka.producer.async.ProducerSendThread) [2013-01-11 09:41:07,520] DEBUG Got ping response for sessionid: 0x13c012a599601aa after 144ms (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:41:09,520] DEBUG Got ping response for sessionid: 0x13c012a599601aa after 144ms (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:41:11,520] DEBUG Got ping response for sessionid: 0x13c012a599601aa after 144ms (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:41:11,871] DEBUG 5002 ms elapsed. Queue time reached. Sending.. (kafka.producer.async.ProducerSendThread) [2013-01-11 09:41:11,871] DEBUG Handling 0 events (kafka.producer.async.ProducerSendThread) [2013-01-11 09:41:13,520] DEBUG Got ping response for sessionid: 0x13c012a599601aa after 144ms (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:41:15,520] DEBUG Got ping response for sessionid: 0x13c012a599601aa after 144ms (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:41:16,873] DEBUG 5002 ms elapsed. Queue time reached. Sending.. (kafka.producer.async.ProducerSendThread) [2013-01-11 09:41:16,873] DEBUG Handling 0 events (kafka.producer.async.ProducerSendThread) [2013-01-11 09:41:17,521] DEBUG Got ping response for sessionid: 0x13c012a599601aa after 144ms (org.apache.zookeeper.ClientCnxn) [2013-01-11 09:41:19,522] DEBUG Got ping response for sessionid: 0x13c012a599601aa after 144ms (org.apache.zookeeper.ClientCnxn)