EventThread =2014-05-23 07:55:03,221 INFO [Messaging Worker-EventThread] (org.I0Itec.zkclient.ZkClient) - zookeeper state changed (Disconnected) =2014-05-23 07:55:14,844 INFO [Messaging Worker-EventThread] (org.I0Itec.zkclient.ZkClient) - zookeeper state changed (Expired) =2014-05-23 07:55:14,844 DEBUG [Messaging Worker-EventThread] (org.apache.zookeeper.ZooKeeper) - Close called on already closed client =2014-05-23 07:55:14,844 INFO [Messaging Worker-EventThread] (org.apache.zookeeper.ZooKeeper) - Initiating client connection, connectString=mbus0001:2181,mbus0002:2181,mbus0003:2181,mbus0004:2181,mbus0005:2181/kafka8Agg sessionTimeout=60000 watcher=org.I0Itec.zkclient.ZkClient@7f40a84d =2014-05-23 07:55:14,845 INFO [Messaging Worker-EventThread] (org.apache.zookeeper.ClientCnxn) - EventThread shut down =2014-05-23 07:55:40,053 INFO [Messaging Worker-EventThread] (org.I0Itec.zkclient.ZkClient) - zookeeper state changed (SyncConnected)ooo0001 ====================================================================================== SendThread =2014-05-23 07:54:57,622 INFO [Messaging Worker-SendThread(mbus0003:2181)] (org.apache.zookeeper.ClientCnxn) - Client session timed out, have not heard from server in 27460ms for sessionid 0x345f6cac6ed071d, closing socket connection and attempting reconnect =2014-05-23 07:55:14,687 INFO [Messaging Worker-SendThread(mbus0003:2181)] (org.apache.zookeeper.ClientCnxn) - Opening socket connection to server /10.193.212.123:2181 =2014-05-23 07:55:14,693 INFO [Messaging Worker-SendThread(mbus0002:2181)] (org.apache.zookeeper.ClientCnxn) - Client session timed out, have not heard from server in 11472ms for sessionid 0x345f6cac6ed071d, closing socket connection and attempting reconnect =2014-05-23 07:55:14,695 DEBUG [Messaging Worker-SendThread(mbus0002:2181)] (org.apache.zookeeper.ClientCnxnSocketNIO) - Ignoring exception during shutdown input =2014-05-23 07:55:14,695 DEBUG [Messaging Worker-SendThread(mbus0002:2181)] (org.apache.zookeeper.ClientCnxnSocketNIO) - Ignoring exception during shutdown output =2014-05-23 07:55:14,823 INFO [Messaging Worker-SendThread(mbus0002:2181)] (org.apache.zookeeper.ClientCnxn) - Opening socket connection to server /10.193.212.60:2181 =2014-05-23 07:55:14,853 INFO [Messaging Worker-SendThread(mbus0002:2181)] (org.apache.zookeeper.ClientCnxn) - Socket connection established to mbus0002/10.193.212.123:2181, initiating session =2014-05-23 07:55:14,853 DEBUG [Messaging Worker-SendThread(mbus0002:2181)] (org.apache.zookeeper.ClientCnxn) - Session establishment request sent on mbus0002/10.193.212.123:2181 =2014-05-23 07:55:26,951 INFO [Messaging Worker-SendThread(mbus0002:2181)] (org.apache.zookeeper.ClientCnxn) - Client session timed out, have not heard from server in 12098ms for sessionid 0x0, closing socket connection and attempting reconnect =2014-05-23 07:55:40,036 INFO [Messaging Worker-SendThread(mbus0002:2181)] (org.apache.zookeeper.ClientCnxn) - Opening socket connection to server /10.193.212.59:2181 =2014-05-23 07:55:40,038 INFO [Messaging Worker-SendThread(mbus0005:2181)] (org.apache.zookeeper.ClientCnxn) - Socket connection established to mbus0005/10.193.212.59:2181, initiating session =2014-05-23 07:55:40,038 DEBUG [Messaging Worker-SendThread(mbus0005:2181)] (org.apache.zookeeper.ClientCnxn) - Session establishment request sent on mbus0005/10.193.212.59:2181 =2014-05-23 07:55:40,051 INFO [Messaging Worker-SendThread(mbus0005:2181)] (org.apache.zookeeper.ClientCnxn) - Session establishment complete on server mbus0005/10.193.212.59:2181, sessionid = 0x545f6dc6f510757, negotiated timeout = 40000 =2014-05-23 07:55:40,159 DEBUG [Messaging Worker-SendThread(mbus0005:2181)] (org.apache.zookeeper.ClientCnxn) - Reading reply sessionid:0x545f6dc6f510757, packet:: clientPath:null serverPath:null finished:false header:: 2,5 replyHeader:: 2,150396713424,0 request:: '/kafka8Agg/consumers/myconsumergroup/offsets/TopicA/1,#34343138,-1 response:: s{150323880219,150396713424,1400006263131,1400846140147,4418,0,0,0,4,0,150323880219} =2014-05-23 07:55:40,160 DEBUG [Messaging Worker-SendThread(mbus0005:2181)] (org.apache.zookeeper.ClientCnxn) - Reading reply sessionid:0x545f6dc6f510757, packet:: clientPath:null serverPath:null finished:false header:: 3,1 replyHeader:: 3,150396713425,-110 request:: '/kafka8Agg/consumers/myconsumergroup/ids/myconsumergroup_ooo0001-1400815740329-5055aeee,#7b20227061747465726e223a22737461746963222c2022737562736372697074696f6e223a7b202253746f70416c6c5f55532d45415354223a2031207d2c202274696d657374616d70223a2231343030383436313134383435222c202276657273696f6e223a31207d,v{s{31,s{'world,'anyone}}},1 response:: =2014-05-23 07:55:40,180 DEBUG [Messaging Worker-SendThread(mbus0005:2181)] (org.apache.zookeeper.ClientCnxn) - Reading reply sessionid:0x545f6dc6f510757, packet:: clientPath:null serverPath:null finished:false header:: 4,4 replyHeader:: 4,150396713430,0 request:: '/kafka8Agg/consumers/myconsumergroup/ids/myconsumergroup_ooo0001-1400815740329-5055aeee,F response:: #7b20227061747465726e223a22737461746963222c2022737562736372697074696f6e223a7b202253746f70416c6c5f55532d45415354223a2031207d2c202274696d657374616d70223a2231343030383436313134383435222c202276657273696f6e223a31207d,s{150396710990,150396710990,1400846114866,1400846114866,0,0,0,163808312244176699,105,0,150396710990} =2014-05-23 07:55:40,198 DEBUG [Messaging Worker-SendThread(mbus0005:2181)] (org.apache.zookeeper.ClientCnxn) - Reading reply sessionid:0x545f6dc6f510757, packet:: clientPath:null serverPath:null finished:false header:: 5,8 replyHeader:: 5,150396713431,0 request:: '/kafka8Agg/brokers/ids,F response:: v{'3,'2,'1,'10,'7,'6,'5,'4,'9,'8} =2014-05-23 07:55:53,914 DEBUG [Messaging Worker-SendThread(mbus0005:2181)] (org.apache.zookeeper.ClientCnxn) - Reading reply sessionid:0x545f6dc6f510755, packet:: clientPath:null serverPath:null finished:false header:: 15,4 replyHeader:: 15,150396714438,0 request:: '/kafka8Agg/consumers/myconsumergroup/ids/myconsumergroup_ooo0001-1400815740329-5055aeee,F response:: #7b20227061747465726e223a22737461746963222c2022737562736372697074696f6e223a7b202253746f70416c6c5f55532d45415354223a2031207d2c202274696d657374616d70223a2231343030383436313134383435222c202276657273696f6e223a31207d,s{150396710990,150396710990,1400846114866,1400846114866,0,0,0,163808312244176699,105,0,150396710990} =2014-05-23 07:56:07,186 DEBUG [Messaging Worker-SendThread(mbus0005:2181)] (org.apache.zookeeper.ClientCnxn) - Reading reply sessionid:0x545f6dc6f510757, packet:: clientPath:null serverPath:null finished:false header:: 16,4 replyHeader:: 16,150396715646,-101 request:: '/kafka8Agg/consumers/myconsumergroup/ids/myconsumergroup_ooo0001-1400815740329-5055aeee,F response:: * 163808312244176699 = 0x245f6cac692073b ====================================================================================== kafka client log =2014-05-23 07:55:14,845 INFO [ZkClient-EventThread-428-mbus0001:2181,mbus0002:2181,mbus0003:2181,mbus0004:2181,mbus0005:2181/kafka8Agg] (kafka.consumer.ZookeeperConsumerConnector) - [myconsumergroup_ooo0001-1400815740329-5055aeee], ZK expired; release old broker parition ownership; re-register consumer myconsumergroup_ooo0001-1400815740329-5055aeee =2014-05-23 07:55:14,845 INFO [ZkClient-EventThread-428-mbus0001:2181,mbus0002:2181,mbus0003:2181,mbus0004:2181,mbus0005:2181/kafka8Agg] (kafka.consumer.ZookeeperConsumerConnector) - [myconsumergroup_ooo0001-1400815740329-5055aeee], begin registering consumer myconsumergroup_ooo0001-1400815740329-5055aeee in ZK =2014-05-23 07:55:40,186 INFO [ZkClient-EventThread-428-mbus0001:2181,mbus0002:2181,mbus0003:2181,mbus0004:2181,mbus0005:2181/kafka8Agg] (kafka.utils.ZkUtils$) - /consumers/myconsumergroup/ids/myconsumergroup_ooo0001-1400815740329-5055aeee exists with value { "pattern":"static", "subscription":{ "": 1 }, "timestamp":"1400846114845", "version":1 } during connection loss; this is ok =2014-05-23 07:55:40,186 INFO [ZkClient-EventThread-428-mbus0001:2181,mbus0002:2181,mbus0003:2181,mbus0004:2181,mbus0005:2181/kafka8Agg] (kafka.consumer.ZookeeperConsumerConnector) - [myconsumergroup_ooo0001-1400815740329-5055aeee], end registering consumer myconsumergroup_ooo0001-1400815740329-5055aeee in ZK =2014-05-23 07:55:40,186 INFO [ZkClient-EventThread-428-mbus0001:2181,mbus0002:2181,mbus0003:2181,mbus0004:2181,mbus0005:2181/kafka8Agg] (kafka.con sumer.ZookeeperConsumerConnector) - [myconsumergroup_ooo0001-1400815740329-5055aeee], begin rebalancing consumer myconsumergroup_ooo0001-1400815740329-5055aeee try #0 =2014-05-23 07:56:07,198 INFO [ZkClient-EventThread-428-mbus0001:2181,mbus0002:2181,mbus0003:2181,mbus0004:2181,mbus0005:2181/kafka8Agg] (kafka.consumer.ZookeeperConsumerConnector) - [myconsumergroup_ooo0001-1400815740329-5055aeee], exception during rebalance org.I0Itec.zkclient.exception.ZkNoNodeException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /consumers/myconsumergroup/ids/myconsumergroup_ooo0001-1400815740329-5055aeee ====================================================================================== Another thread's SendThread log =2014-05-23 07:55:27,024 DEBUG [Pipe-SendThread(mbus0002:2181)] (org.apache.zookeeper.ClientCnxn) - Reading reply sessionid:0x245f6cac6920715, packet:: clientPath:null serverPath:null finished:false header:: 332,4 replyHeader:: 332,150396711891,0 request:: '/kafka8Agg/consumers/myconsumergroup/ids/myconsumergroup_ooo0001-1400815740329-5055aeee,F response:: #7b20227061747465726e223a22737461746963222c2022737562736372697074696f6e223a7b202253746f70416c6c5f55532d45415354223a2031207d2c202274696d657374616d70223a2231343030383436313134383435222c202276657273696f6e223a31207d,s{150396710990,150396710990,1400846114866,1400846114866,0,0,0,163808312244176699,105,0,150396710990}