2015-09-08T12:52:06.069-04:00 org.apache.zookeeper.ClientCnxn main-SendThread(10.130.64.53:2181) INFO: Client session timed out, have not heard from server in 14069ms for sessionid 0x14b0d7b019018a7, closing socket connection and attempting reconnect 2015-09-08T12:52:06.172-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Disconnected) 2015-09-08T12:52:06.656-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Expired) 2015-09-08T12:52:06.665-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (SyncConnected) 2015-09-08T12:52:06.667-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:52:19.107-04:00 org.apache.zookeeper.ClientCnxn main-SendThread(10.130.64.53:2181) INFO: Client session timed out, have not heard from server in 12407ms for sessionid 0x24b0d7b019d1590, closing socket connection and attempting reconnect 2015-09-08T12:52:19.208-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Disconnected) 2015-09-08T12:52:31.639-04:00 org.apache.zookeeper.ClientCnxn main-SendThread(10.130.64.52:2181) INFO: Client session timed out, have not heard from server in 12432ms for sessionid 0x24b0d7b019d1590, closing socket connection and attempting reconnect 2015-09-08T12:52:47.045-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Expired) 2015-09-08T12:52:47.051-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (SyncConnected) 2015-09-08T12:52:47.144-04:00 kafka.network.Processor kafka-network-thread-9092-2 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:197) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) at kafka.utils.Utils$.read(Utils.scala:380) at kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54) at kafka.network.Processor.read(SocketServer.scala:444) at kafka.network.Processor.run(SocketServer.scala:340) at java.lang.Thread.run(Thread.java:745) 2015-09-08T12:52:47.144-04:00 kafka.network.Processor kafka-network-thread-9092-2 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:197) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) at kafka.utils.Utils$.read(Utils.scala:380) at kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54) at kafka.network.Processor.read(SocketServer.scala:444) at kafka.network.Processor.run(SocketServer.scala:340) at java.lang.Thread.run(Thread.java:745) 2015-09-08T12:52:47.209-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:52:49.212-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #1 2015-09-08T12:52:49.225-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #1 2015-09-08T12:52:51.227-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #2 2015-09-08T12:52:51.242-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #2 2015-09-08T12:52:53.243-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #3 2015-09-08T12:52:53.257-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #3 2015-09-08T12:52:55.258-04:00 org.I0Itec.zkclient.ZkEventThread ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka ERROR: Error handling event ZkEvent[New session event sent to kafka.consumer.ZookeeperConsumerConnector$ZKSessionExpireListener@521cffe8] kafka.common.ConsumerRebalanceFailedException: prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 can't rebalance after 4 retries at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:633) at kafka.consumer.ZookeeperConsumerConnector$ZKSessionExpireListener.handleNewSession(ZookeeperConsumerConnector.scala:487) at org.I0Itec.zkclient.ZkClient$4.run(ZkClient.java:472) at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71) 2015-09-08T12:52:55.260-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:52:55.458-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:52:55.460-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:52:55.731-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:52:55.732-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:52:56.003-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:52:56.003-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:52:56.282-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:53:01.263-04:00 org.apache.zookeeper.ClientCnxn main-SendThread(10.130.64.52:2181) INFO: Client session timed out, have not heard from server in 4972ms for sessionid 0x14b0d7b019018ab, closing socket connection and attempting reconnect 2015-09-08T12:53:01.363-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Disconnected) 2015-09-08T12:53:01.409-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (SyncConnected) 2015-09-08T12:53:16.880-04:00 org.apache.zookeeper.ClientCnxn main-SendThread(10.130.64.53:2181) INFO: Client session timed out, have not heard from server in 15460ms for sessionid 0x14b0d7b019018ab, closing socket connection and attempting reconnect 2015-09-08T12:53:16.981-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Disconnected) 2015-09-08T12:53:17.195-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Expired) 2015-09-08T12:53:17.220-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (SyncConnected) 2015-09-08T12:53:17.229-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:53:28.589-04:00 org.apache.zookeeper.ClientCnxn main-SendThread(10.130.64.54:2181) INFO: Client session timed out, have not heard from server in 11212ms for sessionid 0x34b0d893a6e150a, closing socket connection and attempting reconnect 2015-09-08T12:53:28.690-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Disconnected) 2015-09-08T12:53:43.822-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Expired) 2015-09-08T12:53:43.833-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (SyncConnected) 2015-09-08T12:53:43.920-04:00 kafka.network.Processor kafka-network-thread-9092-1 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T12:53:43.920-04:00 kafka.network.Processor kafka-network-thread-9092-1 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T12:53:43.926-04:00 kafka.network.Processor kafka-network-thread-9092-0 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Connection reset by peer at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T12:53:43.926-04:00 kafka.network.Processor kafka-network-thread-9092-0 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Connection reset by peer at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T12:53:43.966-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:53:45.967-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #1 2015-09-08T12:53:45.977-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #1 2015-09-08T12:53:47.978-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #2 2015-09-08T12:53:47.989-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #2 2015-09-08T12:53:49.990-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #3 2015-09-08T12:53:50.001-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #3 2015-09-08T12:53:52.002-04:00 org.I0Itec.zkclient.ZkEventThread ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka ERROR: Error handling event ZkEvent[New session event sent to kafka.consumer.ZookeeperConsumerConnector$ZKSessionExpireListener@521cffe8] kafka.common.ConsumerRebalanceFailedException: prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 can't rebalance after 4 retries at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:633) at kafka.consumer.ZookeeperConsumerConnector$ZKSessionExpireListener.handleNewSession(ZookeeperConsumerConnector.scala:487) at org.I0Itec.zkclient.ZkClient$4.run(ZkClient.java:472) at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71) 2015-09-08T12:53:52.008-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:53:52.167-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:53:52.167-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:53:52.400-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:53:52.401-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:53:52.637-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:53:52.639-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:53:52.681-04:00 kafka.network.Processor kafka-network-thread-9092-0 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T12:53:52.681-04:00 kafka.network.Processor kafka-network-thread-9092-0 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T12:54:02.664-04:00 org.apache.zookeeper.ClientCnxn main-SendThread(10.130.64.53:2181) INFO: Client session timed out, have not heard from server in 9878ms for sessionid 0x24b0d7b019d1591, closing socket connection and attempting reconnect 2015-09-08T12:54:02.765-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Disconnected) 2015-09-08T12:54:03.085-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Expired) 2015-09-08T12:54:03.090-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (SyncConnected) 2015-09-08T12:54:03.296-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:54:03.297-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:54:03.536-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:54:03.537-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:54:03.779-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:54:03.780-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:54:04.388-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:54:16.818-04:00 org.apache.zookeeper.ClientCnxn main-SendThread(10.130.64.52:2181) INFO: Client session timed out, have not heard from server in 12416ms for sessionid 0x14b0d7b019018ac, closing socket connection and attempting reconnect 2015-09-08T12:54:16.919-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Disconnected) 2015-09-08T12:54:29.680-04:00 org.apache.zookeeper.ClientCnxn main-SendThread(10.130.64.53:2181) INFO: Client session timed out, have not heard from server in 12761ms for sessionid 0x14b0d7b019018ac, closing socket connection and attempting reconnect 2015-09-08T12:54:39.808-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Expired) 2015-09-08T12:54:39.820-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (SyncConnected) 2015-09-08T12:54:39.826-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:54:40.043-04:00 kafka.network.Processor kafka-network-thread-9092-2 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Connection reset by peer at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T12:54:40.043-04:00 kafka.network.Processor kafka-network-thread-9092-2 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Connection reset by peer at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T12:54:40.045-04:00 kafka.network.Processor kafka-network-thread-9092-1 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T12:54:40.045-04:00 kafka.network.Processor kafka-network-thread-9092-1 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T12:54:40.185-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:54:40.188-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:54:40.390-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:54:40.390-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:54:40.590-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:54:52.065-04:00 org.apache.zookeeper.ClientCnxn main-SendThread(10.130.64.53:2181) INFO: Client session timed out, have not heard from server in 11467ms for sessionid 0x24b0d7b019d1592, closing socket connection and attempting reconnect 2015-09-08T12:54:52.166-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Disconnected) 2015-09-08T12:54:52.872-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Expired) 2015-09-08T12:54:52.894-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (SyncConnected) 2015-09-08T12:54:52.902-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:55:04.969-04:00 org.apache.zookeeper.ClientCnxn main-SendThread(10.130.64.54:2181) INFO: Client session timed out, have not heard from server in 12058ms for sessionid 0x34b0d893a6e150b, closing socket connection and attempting reconnect 2015-09-08T12:55:05.069-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Disconnected) 2015-09-08T12:55:17.159-04:00 org.apache.zookeeper.ClientCnxn main-SendThread(10.130.64.52:2181) INFO: Client session timed out, have not heard from server in 12090ms for sessionid 0x34b0d893a6e150b, closing socket connection and attempting reconnect 2015-09-08T12:55:17.611-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Expired) 2015-09-08T12:55:17.632-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (SyncConnected) 2015-09-08T12:55:17.730-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:55:17.751-04:00 kafka.network.Processor kafka-network-thread-9092-2 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T12:55:17.751-04:00 kafka.network.Processor kafka-network-thread-9092-2 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T12:55:17.753-04:00 kafka.network.Processor kafka-network-thread-9092-1 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T12:55:17.753-04:00 kafka.network.Processor kafka-network-thread-9092-1 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T12:55:30.415-04:00 org.apache.zookeeper.ClientCnxn main-SendThread(10.130.64.54:2181) INFO: Client session timed out, have not heard from server in 12686ms for sessionid 0x34b0d893a6e150c, closing socket connection and attempting reconnect 2015-09-08T12:55:30.520-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Disconnected) 2015-09-08T12:55:31.491-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Expired) 2015-09-08T12:55:31.495-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (SyncConnected) 2015-09-08T12:55:32.433-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #1 2015-09-08T12:55:32.451-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #1 2015-09-08T12:55:34.452-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #2 2015-09-08T12:55:34.462-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #2 2015-09-08T12:55:36.463-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #3 2015-09-08T12:55:36.472-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #3 2015-09-08T12:55:38.473-04:00 org.I0Itec.zkclient.ZkEventThread ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka ERROR: Error handling event ZkEvent[New session event sent to kafka.consumer.ZookeeperConsumerConnector$ZKSessionExpireListener@521cffe8] kafka.common.ConsumerRebalanceFailedException: prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 can't rebalance after 4 retries at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:633) at kafka.consumer.ZookeeperConsumerConnector$ZKSessionExpireListener.handleNewSession(ZookeeperConsumerConnector.scala:487) at org.I0Itec.zkclient.ZkClient$4.run(ZkClient.java:472) at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71) 2015-09-08T12:55:38.474-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:55:38.628-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:55:38.629-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:55:38.849-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:55:38.850-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:55:39.046-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:55:39.047-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:55:39.256-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:55:54.125-04:00 org.apache.zookeeper.ClientCnxn main-SendThread(10.130.64.54:2181) INFO: Client session timed out, have not heard from server in 14862ms for sessionid 0x34b0d893a6e150d, closing socket connection and attempting reconnect 2015-09-08T12:55:54.226-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Disconnected) 2015-09-08T12:55:54.577-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Expired) 2015-09-08T12:56:06.274-04:00 org.apache.zookeeper.ClientCnxn main-SendThread(10.130.64.52:2181) INFO: Client session timed out, have not heard from server in 11686ms for sessionid 0x0, closing socket connection and attempting reconnect 2015-09-08T12:56:06.557-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (SyncConnected) 2015-09-08T12:56:06.568-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:56:20.348-04:00 org.apache.zookeeper.ClientCnxn main-SendThread(10.130.64.53:2181) INFO: Client session timed out, have not heard from server in 13729ms for sessionid 0x24b0d7b019d1593, closing socket connection and attempting reconnect 2015-09-08T12:56:20.449-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Disconnected) 2015-09-08T12:56:32.894-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Expired) 2015-09-08T12:56:32.909-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (SyncConnected) 2015-09-08T12:56:32.961-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:56:32.985-04:00 kafka.network.Processor kafka-network-thread-9092-0 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T12:56:32.985-04:00 kafka.network.Processor kafka-network-thread-9092-0 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T12:56:32.986-04:00 kafka.network.Processor kafka-network-thread-9092-2 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T12:56:32.986-04:00 kafka.network.Processor kafka-network-thread-9092-2 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T12:56:35.029-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #1 2015-09-08T12:56:35.042-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #1 2015-09-08T12:56:37.043-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #2 2015-09-08T12:56:37.054-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #2 2015-09-08T12:56:39.054-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #3 2015-09-08T12:56:39.064-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #3 2015-09-08T12:56:41.065-04:00 org.I0Itec.zkclient.ZkEventThread ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka ERROR: Error handling event ZkEvent[New session event sent to kafka.consumer.ZookeeperConsumerConnector$ZKSessionExpireListener@521cffe8] kafka.common.ConsumerRebalanceFailedException: prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 can't rebalance after 4 retries at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:633) at kafka.consumer.ZookeeperConsumerConnector$ZKSessionExpireListener.handleNewSession(ZookeeperConsumerConnector.scala:487) at org.I0Itec.zkclient.ZkClient$4.run(ZkClient.java:472) at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71) 2015-09-08T12:56:41.066-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:56:41.211-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:56:41.214-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:56:52.003-04:00 org.apache.zookeeper.ClientCnxn main-SendThread(10.130.64.54:2181) INFO: Client session timed out, have not heard from server in 10772ms for sessionid 0x34b0d893a6e150e, closing socket connection and attempting reconnect 2015-09-08T12:56:52.103-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Disconnected) 2015-09-08T12:56:52.832-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Expired) 2015-09-08T12:56:52.836-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (SyncConnected) 2015-09-08T12:56:53.024-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:56:55.025-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #1 2015-09-08T12:56:55.035-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #1 2015-09-08T12:56:57.035-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #2 2015-09-08T12:56:57.045-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #2 2015-09-08T12:56:59.046-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #3 2015-09-08T12:56:59.058-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #3 2015-09-08T12:57:01.059-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:57:01.059-04:00 org.I0Itec.zkclient.ZkEventThread ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka ERROR: Error handling event ZkEvent[New session event sent to kafka.consumer.ZookeeperConsumerConnector$ZKSessionExpireListener@521cffe8] kafka.common.ConsumerRebalanceFailedException: prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 can't rebalance after 4 retries at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:633) at kafka.consumer.ZookeeperConsumerConnector$ZKSessionExpireListener.handleNewSession(ZookeeperConsumerConnector.scala:487) at org.I0Itec.zkclient.ZkClient$4.run(ZkClient.java:472) at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71) 2015-09-08T12:57:01.195-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:57:01.196-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:57:01.413-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:57:01.414-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:57:01.636-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:57:01.638-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:57:01.823-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:57:01.824-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:57:01.860-04:00 kafka.network.Processor kafka-network-thread-9092-2 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T12:57:01.860-04:00 kafka.network.Processor kafka-network-thread-9092-2 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T12:57:02.061-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:57:13.246-04:00 org.apache.zookeeper.ClientCnxn main-SendThread(10.130.64.52:2181) INFO: Client session timed out, have not heard from server in 11172ms for sessionid 0x14b0d7b019018ae, closing socket connection and attempting reconnect 2015-09-08T12:57:13.347-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Disconnected) 2015-09-08T12:57:25.640-04:00 org.apache.zookeeper.ClientCnxn main-SendThread(10.130.64.54:2181) INFO: Client session timed out, have not heard from server in 12293ms for sessionid 0x14b0d7b019018ae, closing socket connection and attempting reconnect 2015-09-08T12:57:25.781-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Expired) 2015-09-08T12:57:25.802-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (SyncConnected) 2015-09-08T12:57:25.815-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:57:38.829-04:00 org.apache.zookeeper.ClientCnxn main-SendThread(10.130.64.54:2181) INFO: Client session timed out, have not heard from server in 12880ms for sessionid 0x34b0d893a6e150f, closing socket connection and attempting reconnect 2015-09-08T12:57:38.929-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Disconnected) 2015-09-08T12:57:39.156-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Expired) 2015-09-08T12:57:39.175-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (SyncConnected) 2015-09-08T12:57:51.941-04:00 org.apache.zookeeper.ClientCnxn main-SendThread(10.130.64.52:2181) INFO: Client session timed out, have not heard from server in 12736ms for sessionid 0x14b0d7b019018af, closing socket connection and attempting reconnect 2015-09-08T12:57:52.042-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Disconnected) 2015-09-08T12:58:05.415-04:00 org.apache.zookeeper.ClientCnxn main-SendThread(10.130.64.54:2181) INFO: Client session timed out, have not heard from server in 13373ms for sessionid 0x14b0d7b019018af, closing socket connection and attempting reconnect 2015-09-08T12:58:15.135-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Expired) 2015-09-08T12:58:15.156-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (SyncConnected) 2015-09-08T12:58:15.229-04:00 kafka.network.Processor kafka-network-thread-9092-0 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T12:58:15.229-04:00 kafka.network.Processor kafka-network-thread-9092-0 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T12:58:15.231-04:00 kafka.network.Processor kafka-network-thread-9092-2 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T12:58:15.231-04:00 kafka.network.Processor kafka-network-thread-9092-2 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T12:58:25.195-04:00 org.apache.zookeeper.ClientCnxn main-SendThread(10.130.64.53:2181) INFO: Client session timed out, have not heard from server in 9946ms for sessionid 0x24b0d7b019d1594, closing socket connection and attempting reconnect 2015-09-08T12:58:25.295-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Disconnected) 2015-09-08T12:58:26.174-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Expired) 2015-09-08T12:58:26.179-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (SyncConnected) 2015-09-08T12:58:26.382-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:58:28.382-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #1 2015-09-08T12:58:28.392-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #1 2015-09-08T12:58:30.392-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #2 2015-09-08T12:58:30.400-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #2 2015-09-08T12:58:32.401-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #3 2015-09-08T12:58:32.410-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #3 2015-09-08T12:58:34.411-04:00 org.I0Itec.zkclient.ZkEventThread ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka ERROR: Error handling event ZkEvent[New session event sent to kafka.consumer.ZookeeperConsumerConnector$ZKSessionExpireListener@521cffe8] kafka.common.ConsumerRebalanceFailedException: prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 can't rebalance after 4 retries at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:633) at kafka.consumer.ZookeeperConsumerConnector$ZKSessionExpireListener.handleNewSession(ZookeeperConsumerConnector.scala:487) at org.I0Itec.zkclient.ZkClient$4.run(ZkClient.java:472) at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71) 2015-09-08T12:58:34.413-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:58:34.559-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:58:34.560-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:58:34.796-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:58:34.797-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:58:35.042-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:58:35.044-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:58:35.075-04:00 kafka.network.Processor kafka-network-thread-9092-2 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T12:58:35.075-04:00 kafka.network.Processor kafka-network-thread-9092-2 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T12:58:35.249-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:58:35.249-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:58:46.327-04:00 org.apache.zookeeper.ClientCnxn main-SendThread(10.130.64.52:2181) INFO: Client session timed out, have not heard from server in 11055ms for sessionid 0x14b0d7b019018b0, closing socket connection and attempting reconnect 2015-09-08T12:58:46.428-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Disconnected) 2015-09-08T12:58:46.684-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Expired) 2015-09-08T12:58:46.688-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (SyncConnected) 2015-09-08T12:58:46.997-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:58:46.998-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:58:47.238-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:58:47.240-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:58:47.442-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:58:47.442-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:58:47.677-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:59:00.717-04:00 org.apache.zookeeper.ClientCnxn main-SendThread(10.130.64.52:2181) INFO: Client session timed out, have not heard from server in 13033ms for sessionid 0x14b0d7b019018b1, closing socket connection and attempting reconnect 2015-09-08T12:59:00.818-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Disconnected) 2015-09-08T12:59:12.423-04:00 org.apache.zookeeper.ClientCnxn main-SendThread(10.130.64.54:2181) INFO: Client session timed out, have not heard from server in 11605ms for sessionid 0x14b0d7b019018b1, closing socket connection and attempting reconnect 2015-09-08T12:59:22.297-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Expired) 2015-09-08T12:59:22.306-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (SyncConnected) 2015-09-08T12:59:22.317-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:59:22.550-04:00 kafka.network.Processor kafka-network-thread-9092-1 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Connection reset by peer at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T12:59:22.550-04:00 kafka.network.Processor kafka-network-thread-9092-1 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Connection reset by peer at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T12:59:22.556-04:00 kafka.network.Processor kafka-network-thread-9092-0 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T12:59:22.556-04:00 kafka.network.Processor kafka-network-thread-9092-0 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T12:59:34.887-04:00 org.apache.zookeeper.ClientCnxn main-SendThread(10.130.64.53:2181) INFO: Client session timed out, have not heard from server in 12298ms for sessionid 0x24b0d7b019d1595, closing socket connection and attempting reconnect 2015-09-08T12:59:34.988-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Disconnected) 2015-09-08T12:59:35.881-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Expired) 2015-09-08T12:59:35.885-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (SyncConnected) 2015-09-08T12:59:35.960-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:59:37.961-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #1 2015-09-08T12:59:37.978-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #1 2015-09-08T12:59:39.979-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #2 2015-09-08T12:59:39.988-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #2 2015-09-08T12:59:41.988-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #3 2015-09-08T12:59:41.998-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #3 2015-09-08T12:59:43.998-04:00 org.I0Itec.zkclient.ZkEventThread ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka ERROR: Error handling event ZkEvent[New session event sent to kafka.consumer.ZookeeperConsumerConnector$ZKSessionExpireListener@521cffe8] kafka.common.ConsumerRebalanceFailedException: prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 can't rebalance after 4 retries at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:633) at kafka.consumer.ZookeeperConsumerConnector$ZKSessionExpireListener.handleNewSession(ZookeeperConsumerConnector.scala:487) at org.I0Itec.zkclient.ZkClient$4.run(ZkClient.java:472) at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71) 2015-09-08T12:59:43.999-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:59:44.148-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:59:44.150-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:59:44.380-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:59:44.380-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:59:44.609-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:59:44.611-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:59:44.802-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:59:44.803-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:59:45.056-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:59:56.657-04:00 org.apache.zookeeper.ClientCnxn main-SendThread(10.130.64.52:2181) INFO: Client session timed out, have not heard from server in 11594ms for sessionid 0x14b0d7b019018b2, closing socket connection and attempting reconnect 2015-09-08T12:59:56.757-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Disconnected) 2015-09-08T12:59:56.833-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Expired) 2015-09-08T12:59:56.854-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (SyncConnected) 2015-09-08T12:59:56.945-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:59:57.169-04:00 kafka.network.Processor kafka-network-thread-9092-0 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Connection reset by peer at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T12:59:57.169-04:00 kafka.network.Processor kafka-network-thread-9092-0 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Connection reset by peer at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T12:59:57.173-04:00 kafka.network.Processor kafka-network-thread-9092-1 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T12:59:57.173-04:00 kafka.network.Processor kafka-network-thread-9092-1 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T12:59:57.400-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:59:57.405-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:59:57.635-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:59:57.635-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T12:59:57.872-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:00:10.138-04:00 org.apache.zookeeper.ClientCnxn main-SendThread(10.130.64.54:2181) INFO: Client session timed out, have not heard from server in 12258ms for sessionid 0x34b0d893a6e1510, closing socket connection and attempting reconnect 2015-09-08T13:00:10.239-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Disconnected) 2015-09-08T13:00:10.543-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Expired) 2015-09-08T13:00:10.564-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (SyncConnected) 2015-09-08T13:00:10.655-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:00:10.888-04:00 kafka.network.Processor kafka-network-thread-9092-2 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T13:00:10.888-04:00 kafka.network.Processor kafka-network-thread-9092-2 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T13:00:10.888-04:00 kafka.network.Processor kafka-network-thread-9092-2 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T13:00:10.888-04:00 kafka.network.Processor kafka-network-thread-9092-2 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T13:00:11.122-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:00:11.125-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:00:25.252-04:00 org.apache.zookeeper.ClientCnxn main-SendThread(10.130.64.52:2181) INFO: Client session timed out, have not heard from server in 14112ms for sessionid 0x14b0d7b019018b3, closing socket connection and attempting reconnect 2015-09-08T13:00:25.357-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Disconnected) 2015-09-08T13:00:25.867-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Expired) 2015-09-08T13:00:25.871-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (SyncConnected) 2015-09-08T13:00:25.873-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:00:27.874-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #1 2015-09-08T13:00:28.016-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #1 2015-09-08T13:00:28.017-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:00:28.228-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:00:28.229-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:00:28.410-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:00:28.413-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:00:28.609-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:00:38.726-04:00 org.apache.zookeeper.ClientCnxn main-SendThread(10.130.64.53:2181) INFO: Client session timed out, have not heard from server in 10109ms for sessionid 0x24b0d7b019d1596, closing socket connection and attempting reconnect 2015-09-08T13:00:38.826-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Disconnected) 2015-09-08T13:00:39.830-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Expired) 2015-09-08T13:00:39.849-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (SyncConnected) 2015-09-08T13:00:39.859-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:00:40.083-04:00 kafka.network.Processor kafka-network-thread-9092-0 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Connection reset by peer at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T13:00:40.083-04:00 kafka.network.Processor kafka-network-thread-9092-0 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Connection reset by peer at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T13:00:40.091-04:00 kafka.network.Processor kafka-network-thread-9092-2 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T13:00:40.091-04:00 kafka.network.Processor kafka-network-thread-9092-2 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T13:00:40.292-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:00:40.294-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:00:40.477-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:00:40.478-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:00:50.666-04:00 org.apache.zookeeper.ClientCnxn main-SendThread(10.130.64.53:2181) INFO: Client session timed out, have not heard from server in 10173ms for sessionid 0x24b0d7b019d1597, closing socket connection and attempting reconnect 2015-09-08T13:00:50.682-04:00 kafka.network.Processor kafka-network-thread-9092-0 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Connection reset by peer at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T13:00:50.682-04:00 kafka.network.Processor kafka-network-thread-9092-0 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Connection reset by peer at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T13:00:50.767-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Disconnected) 2015-09-08T13:00:51.121-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Expired) 2015-09-08T13:00:51.125-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (SyncConnected) 2015-09-08T13:00:51.213-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:00:53.214-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #1 2015-09-08T13:00:53.354-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #1 2015-09-08T13:00:53.355-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:00:53.550-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:00:53.552-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:00:53.741-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:00:53.742-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:00:54.027-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:01:08.537-04:00 org.apache.zookeeper.ClientCnxn main-SendThread(10.130.64.54:2181) INFO: Client session timed out, have not heard from server in 14502ms for sessionid 0x34b0d893a6e1511, closing socket connection and attempting reconnect 2015-09-08T13:01:08.637-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Disconnected) 2015-09-08T13:01:08.720-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Expired) 2015-09-08T13:01:08.742-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (SyncConnected) 2015-09-08T13:01:19.666-04:00 org.apache.zookeeper.ClientCnxn main-SendThread(10.130.64.52:2181) INFO: Client session timed out, have not heard from server in 10924ms for sessionid 0x14b0d7b019018b4, closing socket connection and attempting reconnect 2015-09-08T13:01:19.766-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Disconnected) 2015-09-08T13:01:31.833-04:00 org.apache.zookeeper.ClientCnxn main-SendThread(10.130.64.53:2181) INFO: Client session timed out, have not heard from server in 12067ms for sessionid 0x14b0d7b019018b4, closing socket connection and attempting reconnect 2015-09-08T13:01:41.536-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Expired) 2015-09-08T13:01:41.547-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (SyncConnected) 2015-09-08T13:01:41.556-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:01:51.775-04:00 org.apache.zookeeper.ClientCnxn main-SendThread(10.130.64.52:2181) INFO: Client session timed out, have not heard from server in 10159ms for sessionid 0x14b0d7b019018b5, closing socket connection and attempting reconnect 2015-09-08T13:02:05.172-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Disconnected) 2015-09-08T13:02:15.251-04:00 org.apache.zookeeper.ClientCnxn main-SendThread(10.130.64.54:2181) INFO: Client session timed out, have not heard from server in 10079ms for sessionid 0x14b0d7b019018b5, closing socket connection and attempting reconnect 2015-09-08T13:02:38.908-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Expired) 2015-09-08T13:02:38.922-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (SyncConnected) 2015-09-08T13:02:49.316-04:00 org.apache.zookeeper.ClientCnxn main-SendThread(10.130.64.52:2181) INFO: Client session timed out, have not heard from server in 10366ms for sessionid 0x14b0d7b019018b6, closing socket connection and attempting reconnect 2015-09-08T13:03:01.970-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Disconnected) 2015-09-08T13:03:13.517-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Expired) 2015-09-08T13:03:13.527-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (SyncConnected) 2015-09-08T13:03:13.546-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:03:24.611-04:00 org.apache.zookeeper.ClientCnxn main-SendThread(10.130.64.52:2181) INFO: Client session timed out, have not heard from server in 11059ms for sessionid 0x14b0d7b019018b7, closing socket connection and attempting reconnect 2015-09-08T13:03:24.623-04:00 kafka.network.Processor kafka-network-thread-9092-2 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Connection reset by peer at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T13:03:24.623-04:00 kafka.network.Processor kafka-network-thread-9092-2 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Connection reset by peer at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T13:03:24.630-04:00 kafka.network.Processor kafka-network-thread-9092-0 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T13:03:24.630-04:00 kafka.network.Processor kafka-network-thread-9092-0 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T13:03:34.729-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Disconnected) 2015-09-08T13:03:35.363-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Expired) 2015-09-08T13:03:35.369-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (SyncConnected) 2015-09-08T13:03:37.477-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #1 2015-09-08T13:03:37.484-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #1 2015-09-08T13:03:39.485-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #2 2015-09-08T13:03:39.494-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #2 2015-09-08T13:03:41.494-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #3 2015-09-08T13:03:41.503-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #3 2015-09-08T13:03:43.504-04:00 org.I0Itec.zkclient.ZkEventThread ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka ERROR: Error handling event ZkEvent[New session event sent to kafka.consumer.ZookeeperConsumerConnector$ZKSessionExpireListener@521cffe8] kafka.common.ConsumerRebalanceFailedException: prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 can't rebalance after 4 retries at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:633) at kafka.consumer.ZookeeperConsumerConnector$ZKSessionExpireListener.handleNewSession(ZookeeperConsumerConnector.scala:487) at org.I0Itec.zkclient.ZkClient$4.run(ZkClient.java:472) at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71) 2015-09-08T13:03:43.505-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:03:43.645-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:03:43.646-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:03:55.011-04:00 org.apache.zookeeper.ClientCnxn main-SendThread(10.130.64.52:2181) INFO: Client session timed out, have not heard from server in 11342ms for sessionid 0x14b0d7b019018b8, closing socket connection and attempting reconnect 2015-09-08T13:03:55.112-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Disconnected) 2015-09-08T13:03:55.717-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Expired) 2015-09-08T13:03:55.721-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (SyncConnected) 2015-09-08T13:03:55.905-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:03:57.905-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #1 2015-09-08T13:03:57.916-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #1 2015-09-08T13:03:59.917-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #2 2015-09-08T13:03:59.927-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #2 2015-09-08T13:04:01.928-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #3 2015-09-08T13:04:01.941-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #3 2015-09-08T13:04:03.942-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:04:03.943-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor ERROR: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], error during syncedRebalance kafka.common.ConsumerRebalanceFailedException: prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 can't rebalance after 4 retries at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:633) at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anon$1.run(ZookeeperConsumerConnector.scala:551) 2015-09-08T13:04:03.952-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:04:05.953-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #1 2015-09-08T13:04:05.965-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #1 2015-09-08T13:04:07.965-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #2 2015-09-08T13:04:07.974-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #2 2015-09-08T13:04:09.975-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #3 2015-09-08T13:04:09.984-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #3 2015-09-08T13:04:11.984-04:00 org.I0Itec.zkclient.ZkEventThread ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka ERROR: Error handling event ZkEvent[New session event sent to kafka.consumer.ZookeeperConsumerConnector$ZKSessionExpireListener@521cffe8] kafka.common.ConsumerRebalanceFailedException: prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 can't rebalance after 4 retries at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:633) at kafka.consumer.ZookeeperConsumerConnector$ZKSessionExpireListener.handleNewSession(ZookeeperConsumerConnector.scala:487) at org.I0Itec.zkclient.ZkClient$4.run(ZkClient.java:472) at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71) 2015-09-08T13:04:11.986-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:04:12.149-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:04:12.150-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:04:12.411-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:04:12.413-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:04:12.659-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:04:12.662-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:04:12.869-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:04:12.870-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:04:24.163-04:00 org.apache.zookeeper.ClientCnxn main-SendThread(10.130.64.52:2181) INFO: Client session timed out, have not heard from server in 11275ms for sessionid 0x14b0d7b019018b9, closing socket connection and attempting reconnect 2015-09-08T13:04:24.264-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Disconnected) 2015-09-08T13:04:24.394-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Expired) 2015-09-08T13:04:24.418-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (SyncConnected) 2015-09-08T13:04:24.484-04:00 kafka.network.Processor kafka-network-thread-9092-1 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T13:04:24.484-04:00 kafka.network.Processor kafka-network-thread-9092-1 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T13:04:24.485-04:00 kafka.network.Processor kafka-network-thread-9092-1 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T13:04:24.485-04:00 kafka.network.Processor kafka-network-thread-9092-1 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T13:04:24.718-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:04:24.719-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:04:24.963-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:04:24.964-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:04:25.167-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:04:25.167-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:04:36.524-04:00 org.apache.zookeeper.ClientCnxn main-SendThread(10.130.64.52:2181) INFO: Client session timed out, have not heard from server in 11338ms for sessionid 0x14b0d7b019018ba, closing socket connection and attempting reconnect 2015-09-08T13:04:36.625-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Disconnected) 2015-09-08T13:04:37.469-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Expired) 2015-09-08T13:04:37.473-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (SyncConnected) 2015-09-08T13:04:37.705-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:04:37.706-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:04:37.949-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:04:37.951-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:04:38.159-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:04:38.160-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:04:38.380-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:04:52.157-04:00 org.apache.zookeeper.ClientCnxn main-SendThread(10.130.64.52:2181) INFO: Client session timed out, have not heard from server in 13769ms for sessionid 0x14b0d7b019018bb, closing socket connection and attempting reconnect 2015-09-08T13:04:52.258-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Disconnected) 2015-09-08T13:04:53.098-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Expired) 2015-09-08T13:04:53.122-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (SyncConnected) 2015-09-08T13:04:53.211-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:05:05.889-04:00 org.apache.zookeeper.ClientCnxn main-SendThread(10.130.64.54:2181) INFO: Client session timed out, have not heard from server in 12428ms for sessionid 0x34b0d893a6e1512, closing socket connection and attempting reconnect 2015-09-08T13:05:05.898-04:00 kafka.network.Processor kafka-network-thread-9092-1 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T13:05:05.898-04:00 kafka.network.Processor kafka-network-thread-9092-1 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T13:05:05.900-04:00 kafka.network.Processor kafka-network-thread-9092-0 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T13:05:05.900-04:00 kafka.network.Processor kafka-network-thread-9092-0 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T13:05:05.992-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Disconnected) 2015-09-08T13:05:06.329-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Expired) 2015-09-08T13:05:06.333-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (SyncConnected) 2015-09-08T13:05:06.513-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:05:08.514-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #1 2015-09-08T13:05:08.522-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #1 2015-09-08T13:05:10.522-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #2 2015-09-08T13:05:10.532-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #2 2015-09-08T13:05:12.533-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #3 2015-09-08T13:05:12.542-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #3 2015-09-08T13:05:14.543-04:00 org.I0Itec.zkclient.ZkEventThread ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka ERROR: Error handling event ZkEvent[New session event sent to kafka.consumer.ZookeeperConsumerConnector$ZKSessionExpireListener@521cffe8] kafka.common.ConsumerRebalanceFailedException: prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 can't rebalance after 4 retries at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:633) at kafka.consumer.ZookeeperConsumerConnector$ZKSessionExpireListener.handleNewSession(ZookeeperConsumerConnector.scala:487) at org.I0Itec.zkclient.ZkClient$4.run(ZkClient.java:472) at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71) 2015-09-08T13:05:14.544-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:05:14.688-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:05:14.690-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:05:14.910-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:05:14.911-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:05:26.184-04:00 org.apache.zookeeper.ClientCnxn main-SendThread(10.130.64.52:2181) INFO: Client session timed out, have not heard from server in 11257ms for sessionid 0x14b0d7b019018bc, closing socket connection and attempting reconnect 2015-09-08T13:05:26.285-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Disconnected) 2015-09-08T13:05:27.059-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Expired) 2015-09-08T13:05:27.065-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (SyncConnected) 2015-09-08T13:05:27.146-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:05:29.147-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #1 2015-09-08T13:05:29.156-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #1 2015-09-08T13:05:31.157-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #2 2015-09-08T13:05:31.166-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #2 2015-09-08T13:05:33.167-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #3 2015-09-08T13:05:33.176-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #3 2015-09-08T13:05:35.177-04:00 org.I0Itec.zkclient.ZkEventThread ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka ERROR: Error handling event ZkEvent[New session event sent to kafka.consumer.ZookeeperConsumerConnector$ZKSessionExpireListener@521cffe8] kafka.common.ConsumerRebalanceFailedException: prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 can't rebalance after 4 retries at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:633) at kafka.consumer.ZookeeperConsumerConnector$ZKSessionExpireListener.handleNewSession(ZookeeperConsumerConnector.scala:487) at org.I0Itec.zkclient.ZkClient$4.run(ZkClient.java:472) at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71) 2015-09-08T13:05:35.178-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:05:35.333-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:05:35.334-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:05:35.354-04:00 kafka.client.ClientUtils$ prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31-leader-finder-thread WARN: Fetching topic metadata with correlation id 66 for topics [Set(binlogs_mailchimp_us1)] from broker [id:2,host:10.130.64.55,port:9092] failed java.nio.channels.ClosedByInterruptException at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202) at sun.nio.ch.SocketChannelImpl.poll(SocketChannelImpl.java:957) at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:204) at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) at java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385) at kafka.utils.Utils$.read(Utils.scala:380) at kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54) at kafka.network.Receive$class.readCompletely(Transmission.scala:56) at kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29) at kafka.network.BlockingChannel.receive(BlockingChannel.scala:111) at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:75) at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72) at kafka.producer.SyncProducer.send(SyncProducer.scala:113) at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58) at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:93) at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60) 2015-09-08T13:05:35.355-04:00 kafka.client.ClientUtils$ prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31-leader-finder-thread WARN: Fetching topic metadata with correlation id 66 for topics [Set(binlogs_mailchimp_us1)] from broker [id:6,host:10.130.64.141,port:9092] failed java.nio.channels.ClosedChannelException at kafka.network.BlockingChannel.send(BlockingChannel.scala:100) at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73) at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72) at kafka.producer.SyncProducer.send(SyncProducer.scala:113) at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58) at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:93) at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60) 2015-09-08T13:05:35.355-04:00 kafka.client.ClientUtils$ prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31-leader-finder-thread WARN: Fetching topic metadata with correlation id 66 for topics [Set(binlogs_mailchimp_us1)] from broker [id:9,host:10.130.64.140,port:9092] failed java.nio.channels.ClosedChannelException at kafka.network.BlockingChannel.send(BlockingChannel.scala:100) at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73) at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72) at kafka.producer.SyncProducer.send(SyncProducer.scala:113) at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58) at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:93) at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60) 2015-09-08T13:05:35.356-04:00 kafka.client.ClientUtils$ prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31-leader-finder-thread WARN: Fetching topic metadata with correlation id 66 for topics [Set(binlogs_mailchimp_us1)] from broker [id:3,host:10.130.64.57,port:9092] failed java.nio.channels.ClosedChannelException at kafka.network.BlockingChannel.send(BlockingChannel.scala:100) at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73) at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72) at kafka.producer.SyncProducer.send(SyncProducer.scala:113) at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58) at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:93) at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60) 2015-09-08T13:05:35.356-04:00 kafka.client.ClientUtils$ prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31-leader-finder-thread WARN: Fetching topic metadata with correlation id 66 for topics [Set(binlogs_mailchimp_us1)] from broker [id:10,host:10.130.64.143,port:9092] failed java.nio.channels.ClosedChannelException at kafka.network.BlockingChannel.send(BlockingChannel.scala:100) at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73) at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72) at kafka.producer.SyncProducer.send(SyncProducer.scala:113) at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58) at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:93) at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60) 2015-09-08T13:05:35.357-04:00 kafka.client.ClientUtils$ prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31-leader-finder-thread WARN: Fetching topic metadata with correlation id 66 for topics [Set(binlogs_mailchimp_us1)] from broker [id:8,host:10.130.64.142,port:9092] failed java.nio.channels.ClosedChannelException at kafka.network.BlockingChannel.send(BlockingChannel.scala:100) at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73) at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72) at kafka.producer.SyncProducer.send(SyncProducer.scala:113) at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58) at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:93) at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60) 2015-09-08T13:05:35.357-04:00 kafka.client.ClientUtils$ prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31-leader-finder-thread WARN: Fetching topic metadata with correlation id 66 for topics [Set(binlogs_mailchimp_us1)] from broker [id:7,host:10.130.64.139,port:9092] failed java.nio.channels.ClosedChannelException at kafka.network.BlockingChannel.send(BlockingChannel.scala:100) at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73) at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72) at kafka.producer.SyncProducer.send(SyncProducer.scala:113) at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58) at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:93) at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60) 2015-09-08T13:05:35.358-04:00 kafka.client.ClientUtils$ prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31-leader-finder-thread WARN: Fetching topic metadata with correlation id 66 for topics [Set(binlogs_mailchimp_us1)] from broker [id:1,host:10.130.64.40,port:9092] failed java.nio.channels.ClosedChannelException at kafka.network.BlockingChannel.send(BlockingChannel.scala:100) at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73) at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72) at kafka.producer.SyncProducer.send(SyncProducer.scala:113) at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58) at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:93) at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60) 2015-09-08T13:05:35.358-04:00 kafka.client.ClientUtils$ prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31-leader-finder-thread WARN: Fetching topic metadata with correlation id 66 for topics [Set(binlogs_mailchimp_us1)] from broker [id:4,host:10.130.64.56,port:9092] failed java.nio.channels.ClosedChannelException at kafka.network.BlockingChannel.send(BlockingChannel.scala:100) at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73) at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72) at kafka.producer.SyncProducer.send(SyncProducer.scala:113) at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58) at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:93) at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60) 2015-09-08T13:05:35.359-04:00 kafka.client.ClientUtils$ prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31-leader-finder-thread WARN: Fetching topic metadata with correlation id 66 for topics [Set(binlogs_mailchimp_us1)] from broker [id:5,host:10.130.64.138,port:9092] failed java.nio.channels.ClosedChannelException at kafka.network.BlockingChannel.send(BlockingChannel.scala:100) at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73) at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72) at kafka.producer.SyncProducer.send(SyncProducer.scala:113) at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58) at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:93) at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60) 2015-09-08T13:05:35.563-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:05:35.564-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:05:35.751-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:05:35.755-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:05:36.005-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:05:48.689-04:00 org.apache.zookeeper.ClientCnxn main-SendThread(10.130.64.52:2181) INFO: Client session timed out, have not heard from server in 12677ms for sessionid 0x14b0d7b019018bd, closing socket connection and attempting reconnect 2015-09-08T13:05:48.802-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Disconnected) 2015-09-08T13:05:49.256-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Expired) 2015-09-08T13:05:49.280-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (SyncConnected) 2015-09-08T13:05:49.368-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:05:49.635-04:00 kafka.network.Processor kafka-network-thread-9092-1 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T13:05:49.635-04:00 kafka.network.Processor kafka-network-thread-9092-1 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T13:05:49.636-04:00 kafka.network.Processor kafka-network-thread-9092-0 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T13:05:49.636-04:00 kafka.network.Processor kafka-network-thread-9092-0 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T13:05:49.872-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:05:49.874-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:05:50.067-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:05:50.067-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:05:50.296-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:06:01.470-04:00 org.apache.zookeeper.ClientCnxn main-SendThread(10.130.64.54:2181) INFO: Client session timed out, have not heard from server in 11167ms for sessionid 0x34b0d893a6e1513, closing socket connection and attempting reconnect 2015-09-08T13:06:01.571-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Disconnected) 2015-09-08T13:06:02.433-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Expired) 2015-09-08T13:06:02.457-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (SyncConnected) 2015-09-08T13:06:02.546-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:06:02.817-04:00 kafka.network.Processor kafka-network-thread-9092-2 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T13:06:02.817-04:00 kafka.network.Processor kafka-network-thread-9092-0 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T13:06:02.817-04:00 kafka.network.Processor kafka-network-thread-9092-0 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T13:06:02.817-04:00 kafka.network.Processor kafka-network-thread-9092-2 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T13:06:03.038-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:06:03.039-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:06:03.234-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:06:03.234-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:06:14.080-04:00 org.apache.zookeeper.ClientCnxn main-SendThread(10.130.64.54:2181) INFO: Client session timed out, have not heard from server in 10743ms for sessionid 0x34b0d893a6e1514, closing socket connection and attempting reconnect 2015-09-08T13:06:14.181-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Disconnected) 2015-09-08T13:06:14.305-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Expired) 2015-09-08T13:06:14.310-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (SyncConnected) 2015-09-08T13:06:14.317-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:06:16.319-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #1 2015-09-08T13:06:16.471-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #1 2015-09-08T13:06:16.472-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:06:16.510-04:00 kafka.network.Processor kafka-network-thread-9092-2 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T13:06:16.510-04:00 kafka.network.Processor kafka-network-thread-9092-2 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T13:06:16.719-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:06:16.722-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:06:16.922-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:06:16.923-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:06:17.148-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:06:30.428-04:00 org.apache.zookeeper.ClientCnxn main-SendThread(10.130.64.54:2181) INFO: Client session timed out, have not heard from server in 13272ms for sessionid 0x34b0d893a6e1515, closing socket connection and attempting reconnect 2015-09-08T13:06:30.528-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Disconnected) 2015-09-08T13:06:30.646-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Expired) 2015-09-08T13:06:30.669-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (SyncConnected) 2015-09-08T13:06:30.758-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:06:31.018-04:00 kafka.network.Processor kafka-network-thread-9092-1 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Connection reset by peer at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T13:06:31.018-04:00 kafka.network.Processor kafka-network-thread-9092-1 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Connection reset by peer at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T13:06:31.027-04:00 kafka.network.Processor kafka-network-thread-9092-1 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T13:06:31.027-04:00 kafka.network.Processor kafka-network-thread-9092-1 ERROR: Closing socket for /10.130.64.145 because of error java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) 2015-09-08T13:06:31.249-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:06:31.251-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:06:31.428-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:06:31.429-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:06:43.759-04:00 org.apache.zookeeper.ClientCnxn main-SendThread(10.130.64.54:2181) INFO: Client session timed out, have not heard from server in 12217ms for sessionid 0x34b0d893a6e1516, closing socket connection and attempting reconnect 2015-09-08T13:06:43.860-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Disconnected) 2015-09-08T13:06:44.245-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Expired) 2015-09-08T13:06:44.248-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (SyncConnected) 2015-09-08T13:06:44.343-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:06:44.344-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:06:44.533-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:06:44.535-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:06:44.708-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:06:44.708-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:06:44.906-04:00 kafka.consumer.ZookeeperConsumerConnector prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31_watcher_executor INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], end rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:06:50.374-04:00 org.apache.zookeeper.ClientCnxn main-SendThread(10.130.64.53:2181) INFO: Client session timed out, have not heard from server in 5460ms for sessionid 0x24b0d7b019d1598, closing socket connection and attempting reconnect 2015-09-08T13:06:50.475-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Disconnected) 2015-09-08T13:06:50.983-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (SyncConnected) 2015-09-08T13:07:03.443-04:00 org.apache.zookeeper.ClientCnxn main-SendThread(10.130.64.52:2181) INFO: Client session timed out, have not heard from server in 12428ms for sessionid 0x24b0d7b019d1598, closing socket connection and attempting reconnect 2015-09-08T13:07:03.543-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Disconnected) 2015-09-08T13:07:15.080-04:00 org.apache.zookeeper.ClientCnxn main-SendThread(10.130.64.54:2181) INFO: Client session timed out, have not heard from server in 11537ms for sessionid 0x24b0d7b019d1598, closing socket connection and attempting reconnect 2015-09-08T13:07:37.418-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Expired) 2015-09-08T13:07:37.429-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (SyncConnected) 2015-09-08T13:07:47.605-04:00 org.apache.zookeeper.ClientCnxn main-SendThread(10.130.64.53:2181) INFO: Client session timed out, have not heard from server in 10177ms for sessionid 0x24b0d7b019d1599, closing socket connection and attempting reconnect 2015-09-08T13:08:01.080-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Disconnected) 2015-09-08T13:08:10.359-04:00 org.apache.zookeeper.ClientCnxn main-SendThread(10.130.64.54:2181) INFO: Client session timed out, have not heard from server in 9279ms for sessionid 0x24b0d7b019d1599, closing socket connection and attempting reconnect 2015-09-08T13:08:29.593-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Expired) 2015-09-08T13:08:37.056-04:00 org.apache.zookeeper.ClientCnxn main-SendThread(10.130.64.53:2181) INFO: Client session timed out, have not heard from server in 7457ms for sessionid 0x0, closing socket connection and attempting reconnect 2015-09-08T13:09:04.574-04:00 org.apache.zookeeper.ClientCnxn main-SendThread(10.130.64.52:2181) INFO: Client session timed out, have not heard from server in 10727ms for sessionid 0x0, closing socket connection and attempting reconnect 2015-09-08T13:09:31.135-04:00 org.apache.zookeeper.ClientCnxn main-SendThread(10.130.64.54:2181) INFO: Client session timed out, have not heard from server in 9006ms for sessionid 0x0, closing socket connection and attempting reconnect 2015-09-08T13:09:57.626-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (SyncConnected) 2015-09-08T13:09:57.629-04:00 kafka.consumer.ZookeeperConsumerConnector ZkClient-EventThread-28-10.130.64.52:2181,10.130.64.53:2181,10.130.64.54:2181/kafka INFO: [prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31], begin rebalancing consumer prologue-second-stage_prod_us1_app01.c4.prologue.prod.atl01.rsglab.com-1441720196058-fbbfaa31 try #0 2015-09-08T13:10:03.630-04:00 org.apache.zookeeper.ClientCnxn main-SendThread(10.130.64.53:2181) INFO: Client session timed out, have not heard from server in 5997ms for sessionid 0x24b0d7b019d159b, closing socket connection and attempting reconnect 2015-09-08T13:10:12.557-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Disconnected) 2015-09-08T13:10:41.330-04:00 org.apache.zookeeper.ClientCnxn main-SendThread(10.130.64.52:2181) INFO: Client session timed out, have not heard from server in 9596ms for sessionid 0x24b0d7b019d159b, closing socket connection and attempting reconnect 2015-09-08T13:11:33.962-04:00 org.I0Itec.zkclient.ZkClient main-EventThread INFO: zookeeper state changed (Expired) 2015-09-08T13:13:41.655-04:00 org.apache.zookeeper.ClientCnxn main-SendThread(10.130.64.53:2181) INFO: Client session timed out, have not heard from server in 47421ms for sessionid 0x0, closing socket connection and attempting reconnect --------