@4000000055f79c36096e6fb4 2015-09-15 04:18:52,158 [ProducerSendThread-5] INFO kafka.producer.SyncProducer - Connected to kafka01.us-west-2.prod.conviva.com:9092 for producing @4000000055f79c3c1898da24 2015-09-15 04:18:58,412 [ProducerSendThread-5] INFO kafka.producer.SyncProducer - Connected to kafka08.us-west-2.prod.conviva.com:9092 for producing @4000000055f79cd33a3dd01c 2015-09-15 04:21:29,977 [ConsumerService STARTING-SendThread(rccp105-12c.iad1.prod.conviva.com:2181)] INFO org.apache.zookeeper.ClientCnxn - Client session timed out, have not heard from server in 26666ms for sessionid 0x84f0b584f7001da, closing socket connection and attempting reconnect @4000000055f79cd4049c9484 2015-09-15 04:21:30,077 [ConsumerService STARTING-EventThread] INFO org.I0Itec.zkclient.ZkClient - zookeeper state changed (Disconnected) @4000000055f79cd42eefd4bc 2015-09-15 04:21:30,787 [ConsumerService STARTING-SendThread(rccp105-12c.iad1.prod.conviva.com:2181)] INFO org.apache.zookeeper.ClientCnxn - Client session timed out, have not heard from server in 26667ms for sessionid 0x84f0b584f7001de, closing socket connection and attempting reconnect @4000000055f79cd434e95384 2015-09-15 04:21:30,887 [ConsumerService STARTING-EventThread] INFO org.I0Itec.zkclient.ZkClient - zookeeper state changed (Disconnected) @4000000055f79cd52f25734c 2015-09-15 04:21:31,790 [ConsumerService STARTING-SendThread(rccp105-12c.iad1.prod.conviva.com:2181)] INFO org.apache.zookeeper.ClientCnxn - Client session timed out, have not heard from server in 26666ms for sessionid 0x84f0b584f7001dc, closing socket connection and attempting reconnect @4000000055f79cd5351f5b8c 2015-09-15 04:21:31,891 [ConsumerService STARTING-EventThread] INFO org.I0Itec.zkclient.ZkClient - zookeeper state changed (Disconnected) @4000000055f79cd62f322164 2015-09-15 04:21:32,791 [ConsumerService STARTING-SendThread(rccp105-13c.iad1.prod.conviva.com:2181)] INFO org.apache.zookeeper.ClientCnxn - Client session timed out, have not heard from server in 26666ms for sessionid 0x94f0b584f5a01ca, closing socket connection and attempting reconnect @4000000055f79cd6352b80ec 2015-09-15 04:21:32,891 [ConsumerService STARTING-EventThread] INFO org.I0Itec.zkclient.ZkClient - zookeeper state changed (Disconnected) @4000000055f79cdc2f3fd534 2015-09-15 04:21:38,792 [ConsumerService STARTING-SendThread(rccp105-11c.iad1.prod.conviva.com:2181)] INFO org.apache.zookeeper.ClientCnxn - Client session timed out, have not heard from server in 26666ms for sessionid 0x74f0b58513101e7, closing socket connection and attempting reconnect @4000000055f79cdc3538e2b4 2015-09-15 04:21:38,892 [ConsumerService STARTING-EventThread] INFO org.I0Itec.zkclient.ZkClient - zookeeper state changed (Disconnected) @4000000055f79ce105b13cbc 2015-09-15 04:21:43,095 [ConsumerFetcherThread-production-uswest2_kafka01.us-west-2.prod.conviva.com-1441843384564-4438dd8d-0-13] INFO kafka.consumer.SimpleConsumer - Reconnect due to socket error: null @4000000055f79ce105b1c95c 2015-09-15 04:21:43,095 [ConsumerFetcherThread-production-uswest2_kafka01.us-west-2.prod.conviva.com-1441843384564-a04e07b7-0-14] INFO kafka.consumer.SimpleConsumer - Reconnect due to socket error: null @4000000055f79ce106291084 2015-09-15 04:21:43,103 [ConsumerFetcherThread-production-uswest2_kafka01.us-west-2.prod.conviva.com-1441843384564-b1e3f9ab-0-16] INFO kafka.consumer.SimpleConsumer - Reconnect due to socket error: null @4000000055f79ce10812cb9c 2015-09-15 04:21:43,135 [ConsumerFetcherThread-production-uswest2_kafka01.us-west-2.prod.conviva.com-1441843384564-4438dd8d-0-16] INFO kafka.consumer.SimpleConsumer - Reconnect due to socket error: null @4000000055f79ce108c6b634 2015-09-15 04:21:43,147 [ConsumerFetcherThread-production-uswest2_kafka01.us-west-2.prod.conviva.com-1441843384564-e723525e-0-14] INFO kafka.consumer.SimpleConsumer - Reconnect due to socket error: null @4000000055f79ce1093f745c 2015-09-15 04:21:43,155 [ConsumerFetcherThread-production-uswest2_kafka01.us-west-2.prod.conviva.com-1441843384564-b1e3f9ab-0-13] INFO kafka.consumer.SimpleConsumer - Reconnect due to socket error: null @4000000055f79ce10aebc0e4 2015-09-15 04:21:43,183 [ConsumerFetcherThread-production-uswest2_kafka01.us-west-2.prod.conviva.com-1441843384564-1a4f6a5b-0-15] INFO kafka.consumer.SimpleConsumer - Reconnect due to socket error: null @4000000055f79ce10b291804 2015-09-15 04:21:43,187 [ConsumerFetcherThread-production-uswest2_kafka01.us-west-2.prod.conviva.com-1441843384564-a04e07b7-0-15] INFO kafka.consumer.SimpleConsumer - Reconnect due to socket error: null @4000000055f79ce10c1cf5b4 2015-09-15 04:21:43,203 [ConsumerFetcherThread-production-uswest2_kafka01.us-west-2.prod.conviva.com-1441843384564-1a4f6a5b-0-14] INFO kafka.consumer.SimpleConsumer - Reconnect due to socket error: null @4000000055f79ce10c1e16c4 2015-09-15 04:21:43,203 [ConsumerFetcherThread-production-uswest2_kafka01.us-west-2.prod.conviva.com-1441843384564-e723525e-0-15] INFO kafka.consumer.SimpleConsumer - Reconnect due to socket error: null @4000000055f79ced207837bc 2015-09-15 04:21:55,544 [ConsumerService STARTING-SendThread(10.0.8.148:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server 10.0.8.148/10.0.8.148:2181. Will not attempt to authenticate using SASL (Unable to locate a login configuration) @4000000055f79cee35e97da4 2015-09-15 04:21:56,904 [ConsumerService STARTING-SendThread(10.0.8.172:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server 10.0.8.172/10.0.8.172:2181. Will not attempt to authenticate using SASL (Unable to locate a login configuration) @4000000055f79cef05b34ffc 2015-09-15 04:21:57,095 [ConsumerService STARTING-SendThread(10.0.8.148:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server 10.0.8.148/10.0.8.148:2181. Will not attempt to authenticate using SASL (Unable to locate a login configuration) @4000000055f79cf01e0278e4 2015-09-15 04:21:58,503 [ConsumerService STARTING-SendThread(10.0.8.148:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server 10.0.8.148/10.0.8.148:2181. Will not attempt to authenticate using SASL (Unable to locate a login configuration) @4000000055f79cf1212c2a24 2015-09-15 04:21:59,556 [ConsumerService STARTING-SendThread(10.0.8.160:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server 10.0.8.160/10.0.8.160:2181. Will not attempt to authenticate using SASL (Unable to locate a login configuration) @4000000055f79cfa35204204 2015-09-15 04:22:08,891 [ConsumerService STARTING-SendThread(10.0.8.148:2181)] INFO org.apache.zookeeper.ClientCnxn - Client session timed out, have not heard from server in 38814ms for sessionid 0x84f0b584f7001da, closing socket connection and attempting reconnect @4000000055f79cfc0ebcb28c 2015-09-15 04:22:10,247 [ConsumerService STARTING-SendThread(10.0.8.172:2181)] INFO org.apache.zookeeper.ClientCnxn - Client session timed out, have not heard from server in 39360ms for sessionid 0x84f0b584f7001de, closing socket connection and attempting reconnect @4000000055f79cfc1a2e7de4 2015-09-15 04:22:10,439 [ConsumerService STARTING-SendThread(10.0.8.148:2181)] INFO org.apache.zookeeper.ClientCnxn - Client session timed out, have not heard from server in 38548ms for sessionid 0x84f0b584f7001dc, closing socket connection and attempting reconnect @4000000055f79cfd32ab5234 2015-09-15 04:22:11,849 [ConsumerService STARTING-SendThread(10.0.8.148:2181)] INFO org.apache.zookeeper.ClientCnxn - Client session timed out, have not heard from server in 38958ms for sessionid 0x94f0b584f5a01ca, closing socket connection and attempting reconnect @4000000055f79cfe359965bc 2015-09-15 04:22:12,899 [ConsumerService STARTING-SendThread(10.0.8.160:2181)] INFO org.apache.zookeeper.ClientCnxn - Client session timed out, have not heard from server in 34007ms for sessionid 0x74f0b58513101e7, closing socket connection and attempting reconnect @4000000055f79d0f045fca04 2015-09-15 04:22:29,073 [ConsumerService STARTING-SendThread(10.0.8.172:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server 10.0.8.172/10.0.8.172:2181. Will not attempt to authenticate using SASL (Unable to locate a login configuration) @4000000055f79d101ca5bf74 2015-09-15 04:22:30,480 [ConsumerService STARTING-SendThread(10.0.8.148:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server 10.0.8.148/10.0.8.148:2181. Will not attempt to authenticate using SASL (Unable to locate a login configuration) @4000000055f79d1117547344 2015-09-15 04:22:31,391 [ConsumerService STARTING-SendThread(10.0.8.172:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server 10.0.8.172/10.0.8.172:2181. Will not attempt to authenticate using SASL (Unable to locate a login configuration) @4000000055f79d1211fc743c 2015-09-15 04:22:32,301 [ConsumerService STARTING-SendThread(10.0.8.160:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server 10.0.8.160/10.0.8.160:2181. Will not attempt to authenticate using SASL (Unable to locate a login configuration) @4000000055f79d130e828404 2015-09-15 04:22:33,243 [ConsumerService STARTING-SendThread(10.0.8.172:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server 10.0.8.172/10.0.8.172:2181. Will not attempt to authenticate using SASL (Unable to locate a login configuration) @4000000055f79d1c193b01dc 2015-09-15 04:22:42,423 [ConsumerService STARTING-SendThread(10.0.8.172:2181)] INFO org.apache.zookeeper.ClientCnxn - Client session timed out, have not heard from server in 33432ms for sessionid 0x84f0b584f7001da, closing socket connection and attempting reconnect @4000000055f79d1d2f2def2c 2015-09-15 04:22:43,791 [ConsumerService STARTING-SendThread(rccp105-12c.iad1.prod.conviva.com:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server rccp105-12c.iad1.prod.conviva.com/10.0.8.160:2181. Will not attempt to authenticate using SASL (Unable to locate a login configuration) @4000000055f79d1d314d8f24 2015-09-15 04:22:43,827 [ConsumerService STARTING-SendThread(10.0.8.148:2181)] INFO org.apache.zookeeper.ClientCnxn - Client session timed out, have not heard from server in 33480ms for sessionid 0x84f0b584f7001de, closing socket connection and attempting reconnect @4000000055f79d1e2bd31014 2015-09-15 04:22:44,735 [ConsumerService STARTING-SendThread(10.0.8.172:2181)] INFO org.apache.zookeeper.ClientCnxn - Client session timed out, have not heard from server in 34196ms for sessionid 0x84f0b584f7001dc, closing socket connection and attempting reconnect @4000000055f79d1f2661a924 2015-09-15 04:22:45,643 [ConsumerService STARTING-SendThread(10.0.8.160:2181)] INFO org.apache.zookeeper.ClientCnxn - Client session timed out, have not heard from server in 33693ms for sessionid 0x94f0b584f5a01ca, closing socket connection and attempting reconnect @4000000055f79d1f292ddccc 2015-09-15 04:22:45,690 [ConsumerService STARTING-SendThread(rccp105-12c.iad1.prod.conviva.com:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server rccp105-12c.iad1.prod.conviva.com/10.0.8.160:2181. Will not attempt to authenticate using SASL (Unable to locate a login configuration) @4000000055f79d2006f97044 2015-09-15 04:22:46,116 [ConsumerService STARTING-SendThread(rccp105-12c.iad1.prod.conviva.com:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server rccp105-12c.iad1.prod.conviva.com/10.0.8.160:2181. Will not attempt to authenticate using SASL (Unable to locate a login configuration) @4000000055f79d2023233b4c 2015-09-15 04:22:46,589 [ConsumerService STARTING-SendThread(10.0.8.172:2181)] INFO org.apache.zookeeper.ClientCnxn - Client session timed out, have not heard from server in 33590ms for sessionid 0x74f0b58513101e7, closing socket connection and attempting reconnect @4000000055f79d2121b714a4 2015-09-15 04:22:47,565 [ConsumerService STARTING-SendThread(rccp105-13c.iad1.prod.conviva.com:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server rccp105-13c.iad1.prod.conviva.com/10.0.8.172:2181. Will not attempt to authenticate using SASL (Unable to locate a login configuration) @4000000055f79d213b7fb274 2015-09-15 04:22:47,998 [ConsumerService STARTING-SendThread(rccp105-11c.iad1.prod.conviva.com:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server rccp105-11c.iad1.prod.conviva.com/10.0.8.148:2181. Will not attempt to authenticate using SASL (Unable to locate a login configuration) @4000000055f79d2b07c46934 2015-09-15 04:22:57,130 [ConsumerService STARTING-SendThread(rccp105-12c.iad1.prod.conviva.com:2181)] INFO org.apache.zookeeper.ClientCnxn - Client session timed out, have not heard from server in 14607ms for sessionid 0x84f0b584f7001da, closing socket connection and attempting reconnect @4000000055f79d2b2b34c4a4 2015-09-15 04:22:57,724 [ConsumerService STARTING-SendThread(10.0.8.148:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server 10.0.8.148/10.0.8.148:2181. Will not attempt to authenticate using SASL (Unable to locate a login configuration) @4000000055f79d2c2fb7b4b4 2015-09-15 04:22:58,800 [ConsumerService STARTING-SendThread(10.0.8.148:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established to 10.0.8.148/10.0.8.148:2181, initiating session @4000000055f79d2c3469619c 2015-09-15 04:22:58,879 [ConsumerService STARTING-SendThread(10.0.8.148:2181)] WARN org.apache.zookeeper.ClientCnxnSocket - Connected to an old server; r-o mode will be unavailable @4000000055f79d2c34696584 2015-09-15 04:22:58,879 [ConsumerService STARTING-EventThread] INFO org.I0Itec.zkclient.ZkClient - zookeeper state changed (Expired) @4000000055f79d2c350251f4 2015-09-15 04:22:58,889 [ConsumerService STARTING-EventThread] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=rccp105-11c.iad1.prod.conviva.com:2181,rccp105-12c.iad1.prod.conviva.com:2181,rccp105-13c.iad1.prod.conviva.com:2181 sessionTimeout=40000 watcher=org.I0Itec.zkclient.ZkClient@3ddcc4cd @4000000055f79d2c359961d4 2015-09-15 04:22:58,899 [ConsumerService STARTING-SendThread(10.0.8.148:2181)] INFO org.apache.zookeeper.ClientCnxn - Unable to reconnect to ZooKeeper service, session 0x84f0b584f7001da has expired, closing socket connection @4000000055f79d2d02198b54 2015-09-15 04:22:59,035 [ConsumerService STARTING-SendThread(rccp105-12c.iad1.prod.conviva.com:2181)] INFO org.apache.zookeeper.ClientCnxn - Client session timed out, have not heard from server in 15108ms for sessionid 0x84f0b584f7001de, closing socket connection and attempting reconnect @4000000055f79d2d1b9cd324 2015-09-15 04:22:59,463 [ConsumerService STARTING-SendThread(rccp105-12c.iad1.prod.conviva.com:2181)] INFO org.apache.zookeeper.ClientCnxn - Client session timed out, have not heard from server in 14628ms for sessionid 0x84f0b584f7001dc, closing socket connection and attempting reconnect @4000000055f79d2d322ccf7c 2015-09-15 04:22:59,841 [ConsumerService STARTING-SendThread(10.0.8.172:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server 10.0.8.172/10.0.8.172:2181. Will not attempt to authenticate using SASL (Unable to locate a login configuration) @4000000055f79d2d36c9cb84 2015-09-15 04:22:59,919 [ConsumerService STARTING-SendThread(10.0.8.172:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established to 10.0.8.172/10.0.8.172:2181, initiating session @4000000055f79d2d3b7a3fec 2015-09-15 04:22:59,997 [ConsumerService STARTING-SendThread(10.0.8.172:2181)] WARN org.apache.zookeeper.ClientCnxnSocket - Connected to an old server; r-o mode will be unavailable @4000000055f79d2d3b7bab34 2015-09-15 04:22:59,997 [ConsumerService STARTING-SendThread(10.0.8.172:2181)] INFO org.apache.zookeeper.ClientCnxn - Unable to reconnect to ZooKeeper service, session 0x84f0b584f7001de has expired, closing socket connection @4000000055f79d2d3b7bed9c 2015-09-15 04:22:59,997 [ConsumerService STARTING-EventThread] INFO org.I0Itec.zkclient.ZkClient - zookeeper state changed (Expired) @4000000055f79d2d3b7bf56c 2015-09-15 04:22:59,997 [ConsumerService STARTING-EventThread] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=rccp105-11c.iad1.prod.conviva.com:2181,rccp105-12c.iad1.prod.conviva.com:2181,rccp105-13c.iad1.prod.conviva.com:2181 sessionTimeout=40000 watcher=org.I0Itec.zkclient.ZkClient@391da0 @4000000055f79d2e0437b44c 2015-09-15 04:23:00,070 [ConsumerService STARTING-SendThread(10.0.8.148:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server 10.0.8.148/10.0.8.148:2181. Will not attempt to authenticate using SASL (Unable to locate a login configuration) @4000000055f79d2e08d13d84 2015-09-15 04:23:00,147 [ConsumerService STARTING-SendThread(10.0.8.148:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established to 10.0.8.148/10.0.8.148:2181, initiating session @4000000055f79d2e0d825214 2015-09-15 04:23:00,226 [ConsumerService STARTING-SendThread(10.0.8.148:2181)] WARN org.apache.zookeeper.ClientCnxnSocket - Connected to an old server; r-o mode will be unavailable @4000000055f79d2e0d8259e4 2015-09-15 04:23:00,226 [ConsumerService STARTING-SendThread(10.0.8.148:2181)] INFO org.apache.zookeeper.ClientCnxn - Unable to reconnect to ZooKeeper service, session 0x84f0b584f7001dc has expired, closing socket connection @4000000055f79d2e0d83e46c 2015-09-15 04:23:00,226 [ConsumerService STARTING-EventThread] INFO org.I0Itec.zkclient.ZkClient - zookeeper state changed (Expired) @4000000055f79d2e0d8480ac 2015-09-15 04:23:00,226 [ConsumerService STARTING-EventThread] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=rccp105-11c.iad1.prod.conviva.com:2181,rccp105-12c.iad1.prod.conviva.com:2181,rccp105-13c.iad1.prod.conviva.com:2181 sessionTimeout=40000 watcher=org.I0Itec.zkclient.ZkClient@683c9314 @4000000055f79d2e365051dc 2015-09-15 04:23:00,911 [ConsumerService STARTING-SendThread(rccp105-13c.iad1.prod.conviva.com:2181)] INFO org.apache.zookeeper.ClientCnxn - Client session timed out, have not heard from server in 15167ms for sessionid 0x94f0b584f5a01ca, closing socket connection and attempting reconnect @4000000055f79d2f14756764 2015-09-15 04:23:01,343 [ConsumerService STARTING-SendThread(rccp105-11c.iad1.prod.conviva.com:2181)] INFO org.apache.zookeeper.ClientCnxn - Client session timed out, have not heard from server in 14654ms for sessionid 0x74f0b58513101e7, closing socket connection and attempting reconnect @4000000055f79d2f1d12c984 2015-09-15 04:23:01,487 [ConsumerService STARTING-SendThread(10.0.8.160:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server 10.0.8.160/10.0.8.160:2181. Will not attempt to authenticate using SASL (Unable to locate a login configuration) @4000000055f79d2f21b9915c 2015-09-15 04:23:01,565 [ConsumerService STARTING-SendThread(10.0.8.160:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established to 10.0.8.160/10.0.8.160:2181, initiating session @4000000055f79d2f2664ed14 2015-09-15 04:23:01,644 [ConsumerService STARTING-SendThread(10.0.8.160:2181)] WARN org.apache.zookeeper.ClientCnxnSocket - Connected to an old server; r-o mode will be unavailable @4000000055f79d2f2666b61c 2015-09-15 04:23:01,644 [ConsumerService STARTING-EventThread] INFO org.I0Itec.zkclient.ZkClient - zookeeper state changed (Expired) @4000000055f79d2f2666bdec 2015-09-15 04:23:01,644 [ConsumerService STARTING-EventThread] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=rccp105-11c.iad1.prod.conviva.com:2181,rccp105-12c.iad1.prod.conviva.com:2181,rccp105-13c.iad1.prod.conviva.com:2181 sessionTimeout=40000 watcher=org.I0Itec.zkclient.ZkClient@1ac7057c @4000000055f79d2f2666dd2c 2015-09-15 04:23:01,644 [ConsumerService STARTING-SendThread(10.0.8.160:2181)] INFO org.apache.zookeeper.ClientCnxn - Unable to reconnect to ZooKeeper service, session 0x74f0b58513101e7 has expired, closing socket connection @4000000055f79d2f268bab0c 2015-09-15 04:23:01,646 [ConsumerService STARTING-SendThread(10.0.8.148:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server 10.0.8.148/10.0.8.148:2181. Will not attempt to authenticate using SASL (Unable to locate a login configuration) @4000000055f79d2f2b276e94 2015-09-15 04:23:01,723 [ConsumerService STARTING-SendThread(10.0.8.148:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established to 10.0.8.148/10.0.8.148:2181, initiating session @4000000055f79d2f2fdc8294 2015-09-15 04:23:01,802 [ConsumerService STARTING-SendThread(10.0.8.148:2181)] WARN org.apache.zookeeper.ClientCnxnSocket - Connected to an old server; r-o mode will be unavailable @4000000055f79d2f2fdd45e4 2015-09-15 04:23:01,802 [ConsumerService STARTING-EventThread] INFO org.I0Itec.zkclient.ZkClient - zookeeper state changed (Expired) @4000000055f79d2f2fdd6524 2015-09-15 04:23:01,802 [ConsumerService STARTING-SendThread(10.0.8.148:2181)] INFO org.apache.zookeeper.ClientCnxn - Unable to reconnect to ZooKeeper service, session 0x94f0b584f5a01ca has expired, closing socket connection @4000000055f79d2f2fde1104 2015-09-15 04:23:01,803 [ConsumerService STARTING-EventThread] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=rccp105-11c.iad1.prod.conviva.com:2181,rccp105-12c.iad1.prod.conviva.com:2181,rccp105-13c.iad1.prod.conviva.com:2181 sessionTimeout=40000 watcher=org.I0Itec.zkclient.ZkClient@25c7f37d @4000000055f79d3116c2d9d4 2015-09-15 04:23:03,380 [ConsumerService STARTING-EventThread] ERROR org.apache.zookeeper.ClientCnxn - Error while calling watcher @4000000055f79d3116c2e1a4 java.lang.RuntimeException: Exception while restarting zk client @4000000055f79d3116c2e58c at org.I0Itec.zkclient.ZkClient.processStateChanged(ZkClient.java:462) ~[mirror-maker.jar:na] @4000000055f79d3116c2e974 at org.I0Itec.zkclient.ZkClient.process(ZkClient.java:368) ~[mirror-maker.jar:na] @4000000055f79d3116c2e974 at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519) [mirror-maker.jar:na] @4000000055f79d3116c3146c at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495) [mirror-maker.jar:na] @4000000055f79d3116c31854 Caused by: org.I0Itec.zkclient.exception.ZkException: Unable to connect to rccp105-11c.iad1.prod.conviva.com:2181,rccp105-12c.iad1.prod.conviva.com:2181,rccp105-13c.iad1.prod.conviva.com:2181 @4000000055f79d3116c31c3c at org.I0Itec.zkclient.ZkConnection.connect(ZkConnection.java:66) ~[mirror-maker.jar:na] @4000000055f79d3116c327f4 at org.I0Itec.zkclient.ZkClient.reconnect(ZkClient.java:939) ~[mirror-maker.jar:na] @4000000055f79d3116c32bdc at org.I0Itec.zkclient.ZkClient.processStateChanged(ZkClient.java:458) ~[mirror-maker.jar:na] @4000000055f79d3116c3434c ... 3 common frames omitted @4000000055f79d3116c34734 Caused by: java.net.UnknownHostException: rccp105-11c.iad1.prod.conviva.com @4000000055f79d3116c34734 at java.net.InetAddress.getAllByName0(InetAddress.java:1158) ~[na:1.6.0_26] @4000000055f79d3116c34b1c at java.net.InetAddress.getAllByName(InetAddress.java:1084) ~[na:1.6.0_26] @4000000055f79d3116c34f04 at java.net.InetAddress.getAllByName(InetAddress.java:1020) ~[na:1.6.0_26] @4000000055f79d3116c35abc at org.apache.zookeeper.client.StaticHostProvider.(StaticHostProvider.java:60) ~[mirror-maker.jar:na] @4000000055f79d3116c35ea4 at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:445) ~[mirror-maker.jar:na] @4000000055f79d3116c3628c at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:380) ~[mirror-maker.jar:na] @4000000055f79d3116c3628c at org.I0Itec.zkclient.ZkConnection.connect(ZkConnection.java:64) ~[mirror-maker.jar:na] @4000000055f79d3116c36e44 ... 5 common frames omitted @4000000055f79d3116c3722c 2015-09-15 04:23:03,380 [ConsumerService STARTING-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down @4000000055f79d3116c381cc 2015-09-15 04:23:03,380 [ConsumerService STARTING-EventThread] ERROR org.apache.zookeeper.ClientCnxn - Error while calling watcher @4000000055f79d3116c385b4 java.lang.RuntimeException: Exception while restarting zk client @4000000055f79d3116c3899c at org.I0Itec.zkclient.ZkClient.processStateChanged(ZkClient.java:462) ~[mirror-maker.jar:na] @4000000055f79d3116c3899c at org.I0Itec.zkclient.ZkClient.process(ZkClient.java:368) ~[mirror-maker.jar:na] @4000000055f79d3116c38d84 at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519) [mirror-maker.jar:na] @4000000055f79d3116c3b0ac at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495) [mirror-maker.jar:na] @4000000055f79d3116c3b494 Caused by: org.I0Itec.zkclient.exception.ZkException: Unable to connect to rccp105-11c.iad1.prod.conviva.com:2181,rccp105-12c.iad1.prod.conviva.com:2181,rccp105-13c.iad1.prod.conviva.com:2181 @4000000055f79d3116c3b87c at org.I0Itec.zkclient.ZkConnection.connect(ZkConnection.java:66) ~[mirror-maker.jar:na] @4000000055f79d3116c3c434 at org.I0Itec.zkclient.ZkClient.reconnect(ZkClient.java:939) ~[mirror-maker.jar:na] @4000000055f79d3116c3c81c at org.I0Itec.zkclient.ZkClient.processStateChanged(ZkClient.java:458) ~[mirror-maker.jar:na] @4000000055f79d3116c3dba4 ... 3 common frames omitted @4000000055f79d3116c3dba4 Caused by: java.net.UnknownHostException: rccp105-11c.iad1.prod.conviva.com @4000000055f79d3116c3df8c at java.net.InetAddress.getAllByName0(InetAddress.java:1158) ~[na:1.6.0_26] @4000000055f79d3116c3e374 at java.net.InetAddress.getAllByName(InetAddress.java:1084) ~[na:1.6.0_26] @4000000055f79d3116c3e374 at java.net.InetAddress.getAllByName(InetAddress.java:1020) ~[na:1.6.0_26] @4000000055f79d3116c3ef2c at org.apache.zookeeper.client.StaticHostProvider.(StaticHostProvider.java:60) ~[mirror-maker.jar:na] @4000000055f79d3116c3f314 at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:445) ~[mirror-maker.jar:na] @4000000055f79d3116c3f6fc at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:380) ~[mirror-maker.jar:na] @4000000055f79d3116c3f6fc at org.I0Itec.zkclient.ZkConnection.connect(ZkConnection.java:64) ~[mirror-maker.jar:na] @4000000055f79d3116c402b4 ... 5 common frames omitted @4000000055f79d3116c402b4 2015-09-15 04:23:03,380 [ConsumerService STARTING-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down @4000000055f79d3116ce767c 2015-09-15 04:23:03,381 [ConsumerFetcherThread-production-uswest2_kafka01.us-west-2.prod.conviva.com-1441843384564-e723525e-0-14] WARN kafka.consumer.ConsumerFetcherThread - [ConsumerFetcherThread-production-uswest2_kafka01.us-west-2.prod.conviva.com-1441843384564-e723525e-0-14], Error in fetch Name: FetchRequest; Version: 0; CorrelationId: 1181183; ClientId: production-uswest2-ConsumerFetcherThread-production-uswest2_kafka01.us-west-2.prod.conviva.com-1441843384564-e723525e-0-14; ReplicaId: -1; MaxWait: 100 ms; MinBytes: 1 bytes; RequestInfo: [GOInfoTable,8] -> PartitionFetchInfo(454570,1000000) @4000000055f79d3116ce95bc java.nio.channels.UnresolvedAddressException: null @4000000055f79d3116ce95bc at sun.nio.ch.Net.checkAddress(Net.java:30) ~[na:1.6.0_26] @4000000055f79d3116ce99a4 at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:480) ~[na:1.6.0_26] @4000000055f79d3116ce99a4 at kafka.network.BlockingChannel.connect(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3116ce99a4 at kafka.consumer.SimpleConsumer.connect(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3116ce9d8c at kafka.consumer.SimpleConsumer.reconnect(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3116cea944 at kafka.consumer.SimpleConsumer.liftedTree1$1(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3116cead2c at kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3116cead2c at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3116ceb4fc at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3116ceb8e4 at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3116ceb8e4 at kafka.metrics.KafkaTimer.time(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3116ced054 at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3116ced43c at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3116ced43c at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3116ced824 at kafka.metrics.KafkaTimer.time(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3116cee3dc at kafka.consumer.SimpleConsumer.fetch(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3116cee3dc at kafka.server.AbstractFetcherThread.processFetchRequest(Unknown Source) [mirror-maker.jar:na] @4000000055f79d3116cee7c4 at kafka.server.AbstractFetcherThread.doWork(Unknown Source) [mirror-maker.jar:na] @4000000055f79d3116cee7c4 at kafka.utils.ShutdownableThread.run(Unknown Source) [mirror-maker.jar:na] @4000000055f79d3116ceef94 2015-09-15 04:23:03,382 [ZkClient-EventThread-47-rccp105-11c.iad1.prod.conviva.com:2181,rccp105-12c.iad1.prod.conviva.com:2181,rccp105-13c.iad1.prod.conviva.com:2181] ERROR org.I0Itec.zkclient.ZkEventThread - Error handling event ZkEvent[Children of /consumers/production-uswest2/ids changed sent to kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener@74cb493d] @4000000055f79d3116cef37c java.lang.NullPointerException: null @4000000055f79d3116cefb4c at org.I0Itec.zkclient.ZkConnection.exists(ZkConnection.java:95) ~[mirror-maker.jar:na] @4000000055f79d3116cefb4c at org.I0Itec.zkclient.ZkClient$3.call(ZkClient.java:439) ~[mirror-maker.jar:na] @4000000055f79d3116ceff34 at org.I0Itec.zkclient.ZkClient$3.call(ZkClient.java:436) ~[mirror-maker.jar:na] @4000000055f79d3116ceff34 at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675) ~[mirror-maker.jar:na] @4000000055f79d3116cf0aec at org.I0Itec.zkclient.ZkClient.exists(ZkClient.java:436) ~[mirror-maker.jar:na] @4000000055f79d3116cf0ed4 at org.I0Itec.zkclient.ZkClient.exists(ZkClient.java:445) ~[mirror-maker.jar:na] @4000000055f79d3116cf0ed4 at org.I0Itec.zkclient.ZkClient$7.run(ZkClient.java:566) ~[mirror-maker.jar:na] @4000000055f79d3116cf12bc at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71) ~[mirror-maker.jar:na] @4000000055f79d3116cf16a4 2015-09-15 04:23:03,382 [ZkClient-EventThread-44-rccp105-11c.iad1.prod.conviva.com:2181,rccp105-12c.iad1.prod.conviva.com:2181,rccp105-13c.iad1.prod.conviva.com:2181] ERROR org.I0Itec.zkclient.ZkEventThread - Error handling event ZkEvent[Children of /consumers/production-uswest2/ids changed sent to kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener@5d05429f] @4000000055f79d3116cf225c java.lang.NullPointerException: null @4000000055f79d3116cf225c at org.I0Itec.zkclient.ZkConnection.exists(ZkConnection.java:95) ~[mirror-maker.jar:na] @4000000055f79d3116cf2a2c at org.I0Itec.zkclient.ZkClient$3.call(ZkClient.java:439) ~[mirror-maker.jar:na] @4000000055f79d3116cf2a2c at org.I0Itec.zkclient.ZkClient$3.call(ZkClient.java:436) ~[mirror-maker.jar:na] @4000000055f79d3116cf31fc at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675) ~[mirror-maker.jar:na] @4000000055f79d3116cf35e4 at org.I0Itec.zkclient.ZkClient.exists(ZkClient.java:436) ~[mirror-maker.jar:na] @4000000055f79d3116cf39cc at org.I0Itec.zkclient.ZkClient.exists(ZkClient.java:445) ~[mirror-maker.jar:na] @4000000055f79d3116cf39cc at org.I0Itec.zkclient.ZkClient$7.run(ZkClient.java:566) ~[mirror-maker.jar:na] @4000000055f79d3116cf419c at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71) ~[mirror-maker.jar:na] @4000000055f79d3116cf419c 2015-09-15 04:23:03,382 [ZkClient-EventThread-46-rccp105-11c.iad1.prod.conviva.com:2181,rccp105-12c.iad1.prod.conviva.com:2181,rccp105-13c.iad1.prod.conviva.com:2181] ERROR org.I0Itec.zkclient.ZkEventThread - Error handling event ZkEvent[Children of /consumers/production-uswest2/ids changed sent to kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener@33deab43] @4000000055f79d3116cf4d54 java.lang.NullPointerException: null @4000000055f79d3116cf4d54 at org.I0Itec.zkclient.ZkConnection.exists(ZkConnection.java:95) ~[mirror-maker.jar:na] @4000000055f79d3116cf60dc at org.I0Itec.zkclient.ZkClient$3.call(ZkClient.java:439) ~[mirror-maker.jar:na] @4000000055f79d3116cf68ac at org.I0Itec.zkclient.ZkClient$3.call(ZkClient.java:436) ~[mirror-maker.jar:na] @4000000055f79d3116cf6c94 at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675) ~[mirror-maker.jar:na] @4000000055f79d3116cf6c94 at org.I0Itec.zkclient.ZkClient.exists(ZkClient.java:436) ~[mirror-maker.jar:na] @4000000055f79d3116cf707c at org.I0Itec.zkclient.ZkClient.exists(ZkClient.java:445) ~[mirror-maker.jar:na] @4000000055f79d3116cf7464 at org.I0Itec.zkclient.ZkClient$7.run(ZkClient.java:566) ~[mirror-maker.jar:na] @4000000055f79d3116cf784c at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71) ~[mirror-maker.jar:na] @4000000055f79d3116cf784c 2015-09-15 04:23:03,382 [ZkClient-EventThread-47-rccp105-11c.iad1.prod.conviva.com:2181,rccp105-12c.iad1.prod.conviva.com:2181,rccp105-13c.iad1.prod.conviva.com:2181] ERROR org.I0Itec.zkclient.ZkEventThread - Error handling event ZkEvent[Data of /brokers/topics/GOInfoTable changed sent to kafka.consumer.ZookeeperConsumerConnector$ZKTopicPartitionChangeListener@4e357fb2] @4000000055f79d3116cf8404 java.lang.NullPointerException: null @4000000055f79d3116cf8404 at org.I0Itec.zkclient.ZkConnection.exists(ZkConnection.java:95) ~[mirror-maker.jar:na] @4000000055f79d3116cf8fbc at org.I0Itec.zkclient.ZkClient$3.call(ZkClient.java:439) ~[mirror-maker.jar:na] @4000000055f79d3116cf8fbc at org.I0Itec.zkclient.ZkClient$3.call(ZkClient.java:436) ~[mirror-maker.jar:na] @4000000055f79d3116cf93a4 at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675) ~[mirror-maker.jar:na] @4000000055f79d3116cf93a4 at org.I0Itec.zkclient.ZkClient.exists(ZkClient.java:436) ~[mirror-maker.jar:na] @4000000055f79d3116cf9b74 at org.I0Itec.zkclient.ZkClient$6.run(ZkClient.java:544) ~[mirror-maker.jar:na] @4000000055f79d3116cf9f5c at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71) ~[mirror-maker.jar:na] @4000000055f79d3116cf9f5c 2015-09-15 04:23:03,382 [ZkClient-EventThread-44-rccp105-11c.iad1.prod.conviva.com:2181,rccp105-12c.iad1.prod.conviva.com:2181,rccp105-13c.iad1.prod.conviva.com:2181] ERROR org.I0Itec.zkclient.ZkEventThread - Error handling event ZkEvent[Data of /brokers/topics/GOInfoTable changed sent to kafka.consumer.ZookeeperConsumerConnector$ZKTopicPartitionChangeListener@459046f] @4000000055f79d3116cfab14 java.lang.NullPointerException: null @4000000055f79d3116cfab14 at org.I0Itec.zkclient.ZkConnection.exists(ZkConnection.java:95) ~[mirror-maker.jar:na] @4000000055f79d3116cfb2e4 at org.I0Itec.zkclient.ZkClient$3.call(ZkClient.java:439) ~[mirror-maker.jar:na] @4000000055f79d3116cfb6cc at org.I0Itec.zkclient.ZkClient$3.call(ZkClient.java:436) ~[mirror-maker.jar:na] @4000000055f79d3116cfb6cc at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675) ~[mirror-maker.jar:na] @4000000055f79d3116cfbab4 at org.I0Itec.zkclient.ZkClient.exists(ZkClient.java:436) ~[mirror-maker.jar:na] @4000000055f79d3116cfbe9c at org.I0Itec.zkclient.ZkClient$6.run(ZkClient.java:544) ~[mirror-maker.jar:na] @4000000055f79d3116cfc284 at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71) ~[mirror-maker.jar:na] @4000000055f79d3116cfc284 2015-09-15 04:23:03,382 [ZkClient-EventThread-45-rccp105-11c.iad1.prod.conviva.com:2181,rccp105-12c.iad1.prod.conviva.com:2181,rccp105-13c.iad1.prod.conviva.com:2181] ERROR org.I0Itec.zkclient.ZkEventThread - Error handling event ZkEvent[Children of /consumers/production-uswest2/ids changed sent to kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener@2d4bd0e2] @4000000055f79d3116cfd224 java.lang.NullPointerException: null @4000000055f79d3116cfd224 at org.I0Itec.zkclient.ZkConnection.exists(ZkConnection.java:95) ~[mirror-maker.jar:na] @4000000055f79d3116cfed7c at org.I0Itec.zkclient.ZkClient$3.call(ZkClient.java:439) ~[mirror-maker.jar:na] @4000000055f79d3116cffd1c at org.I0Itec.zkclient.ZkClient$3.call(ZkClient.java:436) ~[mirror-maker.jar:na] @4000000055f79d3116d00104 at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675) ~[mirror-maker.jar:na] @4000000055f79d3116d00104 at org.I0Itec.zkclient.ZkClient.exists(ZkClient.java:436) ~[mirror-maker.jar:na] @4000000055f79d3116d004ec at org.I0Itec.zkclient.ZkClient.exists(ZkClient.java:445) ~[mirror-maker.jar:na] @4000000055f79d3116d056f4 at org.I0Itec.zkclient.ZkClient$7.run(ZkClient.java:566) ~[mirror-maker.jar:na] @4000000055f79d3116d05adc at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71) ~[mirror-maker.jar:na] @4000000055f79d3116d05ec4 2015-09-15 04:23:03,382 [ZkClient-EventThread-48-rccp105-11c.iad1.prod.conviva.com:2181,rccp105-12c.iad1.prod.conviva.com:2181,rccp105-13c.iad1.prod.conviva.com:2181] ERROR org.I0Itec.zkclient.ZkEventThread - Error handling event ZkEvent[Children of /consumers/production-uswest2/ids changed sent to kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener@77779ce1] @4000000055f79d3116d0724c java.lang.NullPointerException: null @4000000055f79d3116d07634 at org.I0Itec.zkclient.ZkConnection.exists(ZkConnection.java:95) ~[mirror-maker.jar:na] @4000000055f79d3116d08da4 at org.I0Itec.zkclient.ZkClient$3.call(ZkClient.java:439) ~[mirror-maker.jar:na] @4000000055f79d3116d08da4 at org.I0Itec.zkclient.ZkClient$3.call(ZkClient.java:436) ~[mirror-maker.jar:na] @4000000055f79d3116d0918c at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675) ~[mirror-maker.jar:na] @4000000055f79d3116d09574 at org.I0Itec.zkclient.ZkClient.exists(ZkClient.java:436) ~[mirror-maker.jar:na] @4000000055f79d3116d0a12c at org.I0Itec.zkclient.ZkClient.exists(ZkClient.java:445) ~[mirror-maker.jar:na] @4000000055f79d3116d0a514 at org.I0Itec.zkclient.ZkClient$7.run(ZkClient.java:566) ~[mirror-maker.jar:na] @4000000055f79d3116d0a514 at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71) ~[mirror-maker.jar:na] @4000000055f79d3116d0a8fc 2015-09-15 04:23:03,382 [ZkClient-EventThread-45-rccp105-11c.iad1.prod.conviva.com:2181,rccp105-12c.iad1.prod.conviva.com:2181,rccp105-13c.iad1.prod.conviva.com:2181] ERROR org.I0Itec.zkclient.ZkEventThread - Error handling event ZkEvent[Data of /brokers/topics/GOInfoTable changed sent to kafka.consumer.ZookeeperConsumerConnector$ZKTopicPartitionChangeListener@4c7ebca9] @4000000055f79d3116d0bc84 java.lang.NullPointerException: null @4000000055f79d3116d0c06c at org.I0Itec.zkclient.ZkConnection.exists(ZkConnection.java:95) ~[mirror-maker.jar:na] @4000000055f79d3116d0cc24 at org.I0Itec.zkclient.ZkClient$3.call(ZkClient.java:439) ~[mirror-maker.jar:na] @4000000055f79d3116d0cc24 at org.I0Itec.zkclient.ZkClient$3.call(ZkClient.java:436) ~[mirror-maker.jar:na] @4000000055f79d3116d0d00c at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675) ~[mirror-maker.jar:na] @4000000055f79d3116d0d3f4 at org.I0Itec.zkclient.ZkClient.exists(ZkClient.java:436) ~[mirror-maker.jar:na] @4000000055f79d3116d0dfac at org.I0Itec.zkclient.ZkClient$6.run(ZkClient.java:544) ~[mirror-maker.jar:na] @4000000055f79d3116d0e394 at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71) ~[mirror-maker.jar:na] @4000000055f79d3116d0e77c 2015-09-15 04:23:03,382 [ConsumerFetcherThread-production-uswest2_kafka01.us-west-2.prod.conviva.com-1441843384564-4438dd8d-0-13] WARN kafka.consumer.ConsumerFetcherThread - [ConsumerFetcherThread-production-uswest2_kafka01.us-west-2.prod.conviva.com-1441843384564-4438dd8d-0-13], Error in fetch Name: FetchRequest; Version: 0; CorrelationId: 1185513; ClientId: production-uswest2-ConsumerFetcherThread-production-uswest2_kafka01.us-west-2.prod.conviva.com-1441843384564-4438dd8d-0-13; ReplicaId: -1; MaxWait: 100 ms; MinBytes: 1 bytes; RequestInfo: [GOInfoTable,3] -> PartitionFetchInfo(453470,1000000) @4000000055f79d3116d12214 java.nio.channels.UnresolvedAddressException: null @4000000055f79d3116d12214 at sun.nio.ch.Net.checkAddress(Net.java:30) ~[na:1.6.0_26] @4000000055f79d3116d125fc at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:480) ~[na:1.6.0_26] @4000000055f79d3116d129e4 at kafka.network.BlockingChannel.connect(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3116d129e4 at kafka.consumer.SimpleConsumer.connect(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3116d12dcc at kafka.consumer.SimpleConsumer.reconnect(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3116d13984 at kafka.consumer.SimpleConsumer.liftedTree1$1(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3116d13d6c at kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3116d14154 at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3116d14d0c at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3116d15cac at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3116d16094 at kafka.metrics.KafkaTimer.time(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3116d1647c at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3116d17034 at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3116d1741c at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3116d17804 at kafka.metrics.KafkaTimer.time(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3116d17bec at kafka.consumer.SimpleConsumer.fetch(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3116d183bc at kafka.server.AbstractFetcherThread.processFetchRequest(Unknown Source) [mirror-maker.jar:na] @4000000055f79d3116d187a4 at kafka.server.AbstractFetcherThread.doWork(Unknown Source) [mirror-maker.jar:na] @4000000055f79d3116d18b8c at kafka.utils.ShutdownableThread.run(Unknown Source) [mirror-maker.jar:na] @4000000055f79d3116d18b8c 2015-09-15 04:23:03,382 [ZkClient-EventThread-48-rccp105-11c.iad1.prod.conviva.com:2181,rccp105-12c.iad1.prod.conviva.com:2181,rccp105-13c.iad1.prod.conviva.com:2181] ERROR org.I0Itec.zkclient.ZkEventThread - Error handling event ZkEvent[Data of /brokers/topics/GOInfoTable changed sent to kafka.consumer.ZookeeperConsumerConnector$ZKTopicPartitionChangeListener@4ddcb50d] @4000000055f79d3116d1a6e4 java.lang.NullPointerException: null @4000000055f79d3116d1a6e4 at org.I0Itec.zkclient.ZkConnection.exists(ZkConnection.java:95) ~[mirror-maker.jar:na] @4000000055f79d3116d1aeb4 at org.I0Itec.zkclient.ZkClient$3.call(ZkClient.java:439) ~[mirror-maker.jar:na] @4000000055f79d3116d1aeb4 at org.I0Itec.zkclient.ZkClient$3.call(ZkClient.java:436) ~[mirror-maker.jar:na] @4000000055f79d3116d1b29c at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675) ~[mirror-maker.jar:na] @4000000055f79d3116d1b29c at org.I0Itec.zkclient.ZkClient.exists(ZkClient.java:436) ~[mirror-maker.jar:na] @4000000055f79d3116d1ba6c at org.I0Itec.zkclient.ZkClient$6.run(ZkClient.java:544) ~[mirror-maker.jar:na] @4000000055f79d3116d1ba6c at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71) ~[mirror-maker.jar:na] @4000000055f79d311861f374 java.lang.NullPointerException @4000000055f79d311861f75c at org.I0Itec.zkclient.ZkConnection.getChildren(ZkConnection.java:99) @4000000055f79d311864ed2c 2015-09-15 04:23:03,408 [ConsumerFetcherThread-production-uswest2_kafka01.us-west-2.prod.conviva.com-1441843384564-b1e3f9ab-0-13] WARN kafka.consumer.ConsumerFetcherThread - [ConsumerFetcherThread-production-uswest2_kafka01.us-west-2.prod.conviva.com-1441843384564-b1e3f9ab-0-13], Error in fetch Name: FetchRequest; Version: 0; CorrelationId: 1183555; ClientId: production-uswest2-ConsumerFetcherThread-production-uswest2_kafka01.us-west-2.prod.conviva.com-1441843384564-b1e3f9ab-0-13; ReplicaId: -1; MaxWait: 100 ms; MinBytes: 1 bytes; RequestInfo: [GOInfoTable,7] -> PartitionFetchInfo(454476,1000000) @4000000055f79d3118652bac java.nio.channels.UnresolvedAddressException: null @4000000055f79d3118652bac at sun.nio.ch.Net.checkAddress(Net.java:30) ~[na:1.6.0_26] @4000000055f79d3118652f94 at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:480) ~[na:1.6.0_26] @4000000055f79d311865337c at kafka.network.BlockingChannel.connect(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d311865337c at kafka.consumer.SimpleConsumer.connect(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3118653764 at kafka.consumer.SimpleConsumer.reconnect(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3118654aec at kafka.consumer.SimpleConsumer.liftedTree1$1(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3118654ed4 at kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d31186552bc at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3118655e74 at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d311865625c at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3118656644 at kafka.metrics.KafkaTimer.time(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d31186571fc at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d31186571fc at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d31186575e4 at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d31186579cc at kafka.metrics.KafkaTimer.time(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3118658d54 at kafka.consumer.SimpleConsumer.fetch(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d311865913c at kafka.server.AbstractFetcherThread.processFetchRequest(Unknown Source) [mirror-maker.jar:na] @4000000055f79d311865913c at kafka.server.AbstractFetcherThread.doWork(Unknown Source) [mirror-maker.jar:na] @4000000055f79d3118659524 at kafka.utils.ShutdownableThread.run(Unknown Source) [mirror-maker.jar:na] @4000000055f79d311868294c 2015-09-15 04:23:03,409 [ConsumerFetcherThread-production-uswest2_kafka01.us-west-2.prod.conviva.com-1441843384564-b1e3f9ab-0-16] WARN kafka.consumer.ConsumerFetcherThread - [ConsumerFetcherThread-production-uswest2_kafka01.us-west-2.prod.conviva.com-1441843384564-b1e3f9ab-0-16], Error in fetch Name: FetchRequest; Version: 0; CorrelationId: 1192087; ClientId: production-uswest2-ConsumerFetcherThread-production-uswest2_kafka01.us-west-2.prod.conviva.com-1441843384564-b1e3f9ab-0-16; ReplicaId: -1; MaxWait: 100 ms; MinBytes: 1 bytes; RequestInfo: [GOInfoTable,6] -> PartitionFetchInfo(453612,1000000) @4000000055f79d311868488c java.nio.channels.UnresolvedAddressException: null @4000000055f79d3118684c74 at sun.nio.ch.Net.checkAddress(Net.java:30) ~[na:1.6.0_26] @4000000055f79d3118684c74 at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:480) ~[na:1.6.0_26] @4000000055f79d311868505c at kafka.network.BlockingChannel.connect(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3118685444 at kafka.consumer.SimpleConsumer.connect(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d311868582c at kafka.consumer.SimpleConsumer.reconnect(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3118686bb4 at kafka.consumer.SimpleConsumer.liftedTree1$1(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3118686f9c at kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3118687384 at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d31186896ac at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3118689a94 at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3118689e7c at kafka.metrics.KafkaTimer.time(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d311868aa34 at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d311868ae1c at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d311868ae1c at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d311868b204 at kafka.metrics.KafkaTimer.time(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d311868c58c at kafka.consumer.SimpleConsumer.fetch(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d311868c974 at kafka.server.AbstractFetcherThread.processFetchRequest(Unknown Source) [mirror-maker.jar:na] @4000000055f79d311868cd5c at kafka.server.AbstractFetcherThread.doWork(Unknown Source) [mirror-maker.jar:na] @4000000055f79d311868d144 at kafka.utils.ShutdownableThread.run(Unknown Source) [mirror-maker.jar:na] @4000000055f79d311868dcfc 2015-09-15 04:23:03,409 [ConsumerFetcherThread-production-uswest2_kafka01.us-west-2.prod.conviva.com-1441843384564-4438dd8d-0-16] WARN kafka.consumer.ConsumerFetcherThread - [ConsumerFetcherThread-production-uswest2_kafka01.us-west-2.prod.conviva.com-1441843384564-4438dd8d-0-16], Error in fetch Name: FetchRequest; Version: 0; CorrelationId: 1189940; ClientId: production-uswest2-ConsumerFetcherThread-production-uswest2_kafka01.us-west-2.prod.conviva.com-1441843384564-4438dd8d-0-16; ReplicaId: -1; MaxWait: 100 ms; MinBytes: 1 bytes; RequestInfo: [GOInfoTable,2] -> PartitionFetchInfo(453957,1000000) @4000000055f79d311868f854 java.nio.channels.UnresolvedAddressException: null @4000000055f79d311868fc3c at sun.nio.ch.Net.checkAddress(Net.java:30) ~[na:1.6.0_26] @4000000055f79d3118690bdc at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:480) ~[na:1.6.0_26] @4000000055f79d3118690fc4 at kafka.network.BlockingChannel.connect(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3118690fc4 at kafka.consumer.SimpleConsumer.connect(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d31186913ac at kafka.consumer.SimpleConsumer.reconnect(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3118691f64 at kafka.consumer.SimpleConsumer.liftedTree1$1(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d311869234c at kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3118692734 at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d31186932ec at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d31186936d4 at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3118693abc at kafka.metrics.KafkaTimer.time(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3118694674 at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d31186959fc at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3118695de4 at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d31186961cc at kafka.metrics.KafkaTimer.time(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d31186965b4 at kafka.consumer.SimpleConsumer.fetch(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d311869716c at kafka.server.AbstractFetcherThread.processFetchRequest(Unknown Source) [mirror-maker.jar:na] @4000000055f79d3118697554 at kafka.server.AbstractFetcherThread.doWork(Unknown Source) [mirror-maker.jar:na] @4000000055f79d311869793c at kafka.utils.ShutdownableThread.run(Unknown Source) [mirror-maker.jar:na] @4000000055f79d3118697d24 2015-09-15 04:23:03,409 [ConsumerFetcherThread-production-uswest2_kafka01.us-west-2.prod.conviva.com-1441843384564-1a4f6a5b-0-14] WARN kafka.consumer.ConsumerFetcherThread - [ConsumerFetcherThread-production-uswest2_kafka01.us-west-2.prod.conviva.com-1441843384564-1a4f6a5b-0-14], Error in fetch Name: FetchRequest; Version: 0; CorrelationId: 1187461; ClientId: production-uswest2-ConsumerFetcherThread-production-uswest2_kafka01.us-west-2.prod.conviva.com-1441843384564-1a4f6a5b-0-14; ReplicaId: -1; MaxWait: 100 ms; MinBytes: 1 bytes; RequestInfo: [GOInfoTable,0] -> PartitionFetchInfo(454195,1000000) @4000000055f79d311869afec java.nio.channels.UnresolvedAddressException: null @4000000055f79d311869b3d4 at sun.nio.ch.Net.checkAddress(Net.java:30) ~[na:1.6.0_26] @4000000055f79d311869b7bc at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:480) ~[na:1.6.0_26] @4000000055f79d311869bba4 at kafka.network.BlockingChannel.connect(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d311869bba4 at kafka.consumer.SimpleConsumer.connect(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d311869bf8c at kafka.consumer.SimpleConsumer.reconnect(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d311869cb44 at kafka.consumer.SimpleConsumer.liftedTree1$1(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d311869cf2c at kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d311869d314 at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d311869e69c at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d311869ea84 at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d311869ee6c at kafka.metrics.KafkaTimer.time(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d311869fa24 at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d311869fe0c at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d311869fe0c at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d31186a01f4 at kafka.metrics.KafkaTimer.time(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d31186a0dac at kafka.consumer.SimpleConsumer.fetch(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d31186a1194 at kafka.server.AbstractFetcherThread.processFetchRequest(Unknown Source) [mirror-maker.jar:na] @4000000055f79d31186a157c at kafka.server.AbstractFetcherThread.doWork(Unknown Source) [mirror-maker.jar:na] @4000000055f79d31186a1964 at kafka.utils.ShutdownableThread.run(Unknown Source) [mirror-maker.jar:na] @4000000055f79d311872721c at org.I0Itec.zkclient.ZkClient$2.call(ZkClient.java:416) @4000000055f79d3118727604 at org.I0Itec.zkclient.ZkClient$2.call(ZkClient.java:413) @4000000055f79d3118727604 at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675) @4000000055f79d31187279ec at org.I0Itec.zkclient.ZkClient.getChildren(ZkClient.java:413) @4000000055f79d31187279ec at org.I0Itec.zkclient.ZkClient.getChildren(ZkClient.java:409) @4000000055f79d3118727dd4 at kafka.utils.ZkUtils$.getChildrenParentMayNotExist(Unknown Source) @4000000055f79d311872915c at kafka.utils.ZkUtils$.getAllBrokersInCluster(Unknown Source) @4000000055f79d3118729544 at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(Unknown Source) @4000000055f79d3118729544 at kafka.utils.ShutdownableThread.run(Unknown Source) @4000000055f79d311872992c java.lang.NullPointerException @4000000055f79d311872992c at org.I0Itec.zkclient.ZkConnection.getChildren(ZkConnection.java:99) @4000000055f79d311872b484 at org.I0Itec.zkclient.ZkClient$2.call(ZkClient.java:416) @4000000055f79d311872b484 at org.I0Itec.zkclient.ZkClient$2.call(ZkClient.java:413) @4000000055f79d311872b86c at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675) @4000000055f79d311872b86c at org.I0Itec.zkclient.ZkClient.getChildren(ZkClient.java:413) @4000000055f79d311872b86c at org.I0Itec.zkclient.ZkClient.getChildren(ZkClient.java:409) @4000000055f79d311872c03c at kafka.utils.ZkUtils$.getChildrenParentMayNotExist(Unknown Source) @4000000055f79d311872d3c4 at kafka.utils.ZkUtils$.getAllBrokersInCluster(Unknown Source) @4000000055f79d311872d7ac at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(Unknown Source) @4000000055f79d311872db94 at kafka.utils.ShutdownableThread.run(Unknown Source) @4000000055f79d311872eb34 java.lang.NullPointerException @4000000055f79d311873be24 at org.I0Itec.zkclient.ZkConnection.getChildren(ZkConnection.java:99) @4000000055f79d311873fca4 at org.I0Itec.zkclient.ZkClient$2.call(ZkClient.java:416) @4000000055f79d311874373c at org.I0Itec.zkclient.ZkClient$2.call(ZkClient.java:413) @4000000055f79d31187471d4 at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675) @4000000055f79d311874ac6c at org.I0Itec.zkclient.ZkClient.getChildren(ZkClient.java:413) @4000000055f79d311874e31c at org.I0Itec.zkclient.ZkClient.getChildren(ZkClient.java:409) @4000000055f79d311875601c at kafka.utils.ZkUtils$.getChildrenParentMayNotExist(Unknown Source) @4000000055f79d3118756404 at kafka.utils.ZkUtils$.getAllBrokersInCluster(Unknown Source) @4000000055f79d3118757b74 at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(Unknown Source) @4000000055f79d3118758efc at kafka.utils.ShutdownableThread.run(Unknown Source) @4000000055f79d31188cef5c java.lang.NullPointerException @4000000055f79d31188cef5c at org.I0Itec.zkclient.ZkConnection.getChildren(ZkConnection.java:99) @4000000055f79d31188cf344 at org.I0Itec.zkclient.ZkClient$2.call(ZkClient.java:416) @4000000055f79d31188cf72c at org.I0Itec.zkclient.ZkClient$2.call(ZkClient.java:413) @4000000055f79d31188cf72c at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675) @4000000055f79d31188cfb14 at org.I0Itec.zkclient.ZkClient.getChildren(ZkClient.java:413)2015-09-15 04:23:03,411 [production-uswest2_kafka01.us-west-2.prod.conviva.com-1441843384564-4438dd8d-leader-finder-thread] ERROR FATAL kafka.consumer.ConsumerFetcherManager$LeaderFinderThread - [production-uswest2_kafka01.us-west-2.prod.conviva.com-1441843384564-4438dd8d-leader-finder-thread], Encountered NullPointerException. Exiting... @4000000055f79d31188d166c @4000000055f79d31188d1a54 at org.I0Itec.zkclient.ZkClient.getChildren(ZkClient.java:409) @4000000055f79d31188d2224 at kafka.utils.ZkUtils$.getChildrenParentMayNotExist(Unknown Source) @4000000055f79d31188d260c at kafka.utils.ZkUtils$.getAllBrokersInCluster(Unknown Source) @4000000055f79d31188d29f4 at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(Unknown Source) @4000000055f79d31188d2ddc at kafka.utils.ShutdownableThread.run(Unknown Source) @4000000055f79d311892468c 2015-09-15 04:23:03,411 [ZkClient-EventThread-46-rccp105-11c.iad1.prod.conviva.com:2181,rccp105-12c.iad1.prod.conviva.com:2181,rccp105-13c.iad1.prod.conviva.com:2181] ERROR org.I0Itec.zkclient.ZkEventThread - Error handling event ZkEvent[Data of /brokers/topics/GOInfoTable changed sent to kafka.consumer.ZookeeperConsumerConnector$ZKTopicPartitionChangeListener@5fbdd64f] @4000000055f79d3118924e5c java.lang.NullPointerException: null @4000000055f79d3118925244 at org.I0Itec.zkclient.ZkConnection.exists(ZkConnection.java:95) ~[mirror-maker.jar:na] @4000000055f79d31189269b4 at org.I0Itec.zkclient.ZkClient$3.call(ZkClient.java:439) ~[mirror-maker.jar:na] @4000000055f79d3118926d9c at org.I0Itec.zkclient.ZkClient$3.call(ZkClient.java:436) ~[mirror-maker.jar:na] @4000000055f79d3118926d9c at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675) ~[mirror-maker.jar:na] @4000000055f79d3118927184 at org.I0Itec.zkclient.ZkClient.exists(ZkClient.java:436) ~[mirror-maker.jar:na] @4000000055f79d31189294ac at org.I0Itec.zkclient.ZkClient$6.run(ZkClient.java:544) ~[mirror-maker.jar:na] @4000000055f79d3118929894 at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71) ~[mirror-maker.jar:na] @4000000055f79d3118929894 2015-09-15 04:23:03,411 [production-uswest2_kafka01.us-west-2.prod.conviva.com-1441843384564-e723525e-leader-finder-thread] ERROR FATAL kafka.consumer.ConsumerFetcherManager$LeaderFinderThread - [production-uswest2_kafka01.us-west-2.prod.conviva.com-1441843384564-e723525e-leader-finder-thread], Encountered NullPointerException. Exiting... @4000000055f79d311894de9c 2015-09-15 04:23:03,411 [ConsumerFetcherThread-production-uswest2_kafka01.us-west-2.prod.conviva.com-1441843384564-1a4f6a5b-0-15] WARN kafka.consumer.ConsumerFetcherThread - [ConsumerFetcherThread-production-uswest2_kafka01.us-west-2.prod.conviva.com-1441843384564-1a4f6a5b-0-15], Error in fetch Name: FetchRequest; Version: 0; CorrelationId: 1181960; ClientId: production-uswest2-ConsumerFetcherThread-production-uswest2_kafka01.us-west-2.prod.conviva.com-1441843384564-1a4f6a5b-0-15; ReplicaId: -1; MaxWait: 100 ms; MinBytes: 1 bytes; RequestInfo: [GOInfoTable,1] -> PartitionFetchInfo(452607,1000000) @4000000055f79d3118950994 java.nio.channels.UnresolvedAddressException: null @4000000055f79d3118950d7c at sun.nio.ch.Net.checkAddress(Net.java:30) ~[na:1.6.0_26] @4000000055f79d3118952104 at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:480) ~[na:1.6.0_26] @4000000055f79d3118952104 at kafka.network.BlockingChannel.connect(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d31189524ec at kafka.consumer.SimpleConsumer.connect(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d31189528d4 at kafka.consumer.SimpleConsumer.reconnect(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d311895348c at kafka.consumer.SimpleConsumer.liftedTree1$1(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3118953874 at kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3118953c5c at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3118954814 at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3118954bfc at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3118954bfc at kafka.metrics.KafkaTimer.time(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d31189557b4 at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3118956b3c at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3118956b3c at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3118956f24 at kafka.metrics.KafkaTimer.time(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d311895730c at kafka.consumer.SimpleConsumer.fetch(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3118957ec4 at kafka.server.AbstractFetcherThread.processFetchRequest(Unknown Source) [mirror-maker.jar:na] @4000000055f79d3118957ec4 at kafka.server.AbstractFetcherThread.doWork(Unknown Source) [mirror-maker.jar:na] @4000000055f79d31189582ac at kafka.utils.ShutdownableThread.run(Unknown Source) [mirror-maker.jar:na] @4000000055f79d3118958694 2015-09-15 04:23:03,412 [ConsumerService STARTING-EventThread] ERROR org.apache.zookeeper.ClientCnxn - Error while calling watcher @4000000055f79d311895924c java.lang.RuntimeException: Exception while restarting zk client @4000000055f79d311895924c at org.I0Itec.zkclient.ZkClient.processStateChanged(ZkClient.java:462) ~[mirror-maker.jar:na] @4000000055f79d3118959634 at org.I0Itec.zkclient.ZkClient.process(ZkClient.java:368) ~[mirror-maker.jar:na] @4000000055f79d311895bd44 at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519) [mirror-maker.jar:na] @4000000055f79d311895c12c at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495) [mirror-maker.jar:na] @4000000055f79d311895c12c Caused by: org.I0Itec.zkclient.exception.ZkException: Unable to connect to rccp105-11c.iad1.prod.conviva.com:2181,rccp105-12c.iad1.prod.conviva.com:2181,rccp105-13c.iad1.prod.conviva.com:2181 @4000000055f79d311895d0cc at org.I0Itec.zkclient.ZkConnection.connect(ZkConnection.java:66) ~[mirror-maker.jar:na] @4000000055f79d311895d4b4 at org.I0Itec.zkclient.ZkClient.reconnect(ZkClient.java:939) ~[mirror-maker.jar:na] @4000000055f79d311895d89c at org.I0Itec.zkclient.ZkClient.processStateChanged(ZkClient.java:458) ~[mirror-maker.jar:na] @4000000055f79d311895e454 ... 3 common frames omitted @4000000055f79d311895e454 Caused by: java.net.UnknownHostException: rccp105-11c.iad1.prod.conviva.com @4000000055f79d311895e83c at java.net.InetAddress.getAllByName0(InetAddress.java:1158) ~[na:1.6.0_26] @4000000055f79d311895e83c at java.net.InetAddress.getAllByName(InetAddress.java:1084) ~[na:1.6.0_26] @4000000055f79d311895ec24 at java.net.InetAddress.getAllByName(InetAddress.java:1020) ~[na:1.6.0_26] @4000000055f79d311895f7dc at org.apache.zookeeper.client.StaticHostProvider.(StaticHostProvider.java:60) ~[mirror-maker.jar:na] @4000000055f79d3118960b64 at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:445) ~[mirror-maker.jar:na] @4000000055f79d3118960b64 at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:380) ~[mirror-maker.jar:na] @4000000055f79d3118960f4c at org.I0Itec.zkclient.ZkConnection.connect(ZkConnection.java:64) ~[mirror-maker.jar:na] @4000000055f79d3118961334 ... 5 common frames omitted @4000000055f79d3118961eec 2015-09-15 04:23:03,412 [ConsumerFetcherThread-production-uswest2_kafka01.us-west-2.prod.conviva.com-1441843384564-a04e07b7-0-15] WARN kafka.consumer.ConsumerFetcherThread - [ConsumerFetcherThread-production-uswest2_kafka01.us-west-2.prod.conviva.com-1441843384564-a04e07b7-0-15], Error in fetch Name: FetchRequest; Version: 0; CorrelationId: 1192499; ClientId: production-uswest2-ConsumerFetcherThread-production-uswest2_kafka01.us-west-2.prod.conviva.com-1441843384564-a04e07b7-0-15; ReplicaId: -1; MaxWait: 100 ms; MinBytes: 1 bytes; RequestInfo: [GOInfoTable,5] -> PartitionFetchInfo(454420,1000000) @4000000055f79d311896365c java.nio.channels.UnresolvedAddressException: null @4000000055f79d31189645fc at sun.nio.ch.Net.checkAddress(Net.java:30) ~[na:1.6.0_26] @4000000055f79d31189649e4 at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:480) ~[na:1.6.0_26] @4000000055f79d31189649e4 at kafka.network.BlockingChannel.connect(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3118964dcc at kafka.consumer.SimpleConsumer.connect(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d31189651b4 at kafka.consumer.SimpleConsumer.reconnect(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3118965d6c at kafka.consumer.SimpleConsumer.liftedTree1$1(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3118966154 at kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d311896653c at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d31189670f4 at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d31189674dc at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d31189678c4 at kafka.metrics.KafkaTimer.time(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3118968864 at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3118968c4c at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3118969034 at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d311896941c at kafka.metrics.KafkaTimer.time(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d311896b35c at kafka.consumer.SimpleConsumer.fetch(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d311896b744 at kafka.server.AbstractFetcherThread.processFetchRequest(Unknown Source) [mirror-maker.jar:na] @4000000055f79d311896bb2c at kafka.server.AbstractFetcherThread.doWork(Unknown Source) [mirror-maker.jar:na] @4000000055f79d311896bb2c at kafka.utils.ShutdownableThread.run(Unknown Source) [mirror-maker.jar:na] @4000000055f79d311896c6e4 2015-09-15 04:23:03,412 [ConsumerService STARTING-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down @4000000055f79d311896cacc 2015-09-15 04:23:03,412 [ConsumerService STARTING-EventThread] ERROR org.apache.zookeeper.ClientCnxn - Error while calling watcher @4000000055f79d311896d29c java.lang.RuntimeException: Exception while restarting zk client @4000000055f79d311896e23c at org.I0Itec.zkclient.ZkClient.processStateChanged(ZkClient.java:462) ~[mirror-maker.jar:na] @4000000055f79d311896e624 at org.I0Itec.zkclient.ZkClient.process(ZkClient.java:368) ~[mirror-maker.jar:na] @4000000055f79d311896e624 at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519) [mirror-maker.jar:na] @4000000055f79d311896ea0c at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495) [mirror-maker.jar:na] @4000000055f79d311896f5c4 Caused by: org.I0Itec.zkclient.exception.ZkException: Unable to connect to rccp105-11c.iad1.prod.conviva.com:2181,rccp105-12c.iad1.prod.conviva.com:2181,rccp105-13c.iad1.prod.conviva.com:2181 @4000000055f79d311896f9ac at org.I0Itec.zkclient.ZkConnection.connect(ZkConnection.java:66) ~[mirror-maker.jar:na] @4000000055f79d311896fd94 at org.I0Itec.zkclient.ZkClient.reconnect(ZkClient.java:939) ~[mirror-maker.jar:na] @4000000055f79d3118970564 at org.I0Itec.zkclient.ZkClient.processStateChanged(ZkClient.java:458) ~[mirror-maker.jar:na] @4000000055f79d311897094c ... 3 common frames omitted @4000000055f79d3118970d34 Caused by: java.net.UnknownHostException: rccp105-11c.iad1.prod.conviva.com @4000000055f79d3118970d34 at java.net.InetAddress.getAllByName0(InetAddress.java:1158) ~[na:1.6.0_26] @4000000055f79d3118971cd4 at java.net.InetAddress.getAllByName(InetAddress.java:1084) ~[na:1.6.0_26] @4000000055f79d31189720bc at java.net.InetAddress.getAllByName(InetAddress.java:1020) ~[na:1.6.0_26] @4000000055f79d31189724a4 at org.apache.zookeeper.client.StaticHostProvider.(StaticHostProvider.java:60) ~[mirror-maker.jar:na] @4000000055f79d311897288c at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:445) ~[mirror-maker.jar:na] @4000000055f79d3118973444 at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:380) ~[mirror-maker.jar:na] @4000000055f79d311897382c at org.I0Itec.zkclient.ZkConnection.connect(ZkConnection.java:64) ~[mirror-maker.jar:na] @4000000055f79d311897382c ... 5 common frames omitted @4000000055f79d3118973c14 java.lang.NullPointerException2015-09-15 04:23:03,412 [ConsumerService STARTING-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down @4000000055f79d31189747cc @4000000055f79d3118974bb4 at org.I0Itec.zkclient.ZkConnection.getChildren(ZkConnection.java:99) @4000000055f79d3118974bb4 at org.I0Itec.zkclient.ZkClient$2.call(ZkClient.java:416) @4000000055f79d3118974f9c at org.I0Itec.zkclient.ZkClient$2.call(ZkClient.java:413) @4000000055f79d3118975384 at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675) @4000000055f79d3118975b54 at org.I0Itec.zkclient.ZkClient.getChildren(ZkClient.java:413) @4000000055f79d3118976edc at org.I0Itec.zkclient.ZkClient.getChildren(ZkClient.java:409) @4000000055f79d31189772c4 at kafka.utils.ZkUtils$.getChildrenParentMayNotExist(Unknown Source) @4000000055f79d31189772c4 at kafka.utils.ZkUtils$.getAllBrokersInCluster(Unknown Source) @4000000055f79d31189776ac at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(Unknown Source) @4000000055f79d3118977a94 at kafka.utils.ShutdownableThread.run(Unknown Source) @4000000055f79d311898f964 2015-09-15 04:23:03,412 [ConsumerService STARTING-EventThread] ERROR org.apache.zookeeper.ClientCnxn - Error while calling watcher @4000000055f79d311898fd4c java.lang.RuntimeException: Exception while restarting zk client @4000000055f79d3118990134 at org.I0Itec.zkclient.ZkClient.processStateChanged(ZkClient.java:462) ~[mirror-maker.jar:na] @4000000055f79d311899051c at org.I0Itec.zkclient.ZkClient.process(ZkClient.java:368) ~[mirror-maker.jar:na] @4000000055f79d3118990904 at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519) [mirror-maker.jar:na] @4000000055f79d3118991c8c at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495) [mirror-maker.jar:na] @4000000055f79d3118992074 Caused by: org.I0Itec.zkclient.exception.ZkException: Unable to connect to rccp105-11c.iad1.prod.conviva.com:2181,rccp105-12c.iad1.prod.conviva.com:2181,rccp105-13c.iad1.prod.conviva.com:2181 @4000000055f79d311899245c at org.I0Itec.zkclient.ZkConnection.connect(ZkConnection.java:66) ~[mirror-maker.jar:na] @4000000055f79d3118994784 at org.I0Itec.zkclient.ZkClient.reconnect(ZkClient.java:939) ~[mirror-maker.jar:na] @4000000055f79d3118994b6c at org.I0Itec.zkclient.ZkClient.processStateChanged(ZkClient.java:458) ~[mirror-maker.jar:na] @4000000055f79d3118995ef4 ... 3 common frames omitted @4000000055f79d31189962dc Caused by: java.net.UnknownHostException: rccp105-11c.iad1.prod.conviva.com @4000000055f79d31189962dc at java.net.InetAddress.getAllByName0(InetAddress.java:1158) ~[na:1.6.0_26] @4000000055f79d31189966c4 at java.net.InetAddress.getAllByName(InetAddress.java:1084) ~[na:1.6.0_26] @4000000055f79d3118996aac at java.net.InetAddress.getAllByName(InetAddress.java:1020) ~[na:1.6.0_26] @4000000055f79d3118997664 at org.apache.zookeeper.client.StaticHostProvider.(StaticHostProvider.java:60) ~[mirror-maker.jar:na] @4000000055f79d3118997a4c at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:445) ~[mirror-maker.jar:na] @4000000055f79d3118997a4c at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:380) ~[mirror-maker.jar:na] @4000000055f79d3118997e34 at org.I0Itec.zkclient.ZkConnection.connect(ZkConnection.java:64) ~[mirror-maker.jar:na] @4000000055f79d31189989ec ... 5 common frames omitted @4000000055f79d31189989ec 2015-09-15 04:23:03,412 [ConsumerService STARTING-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down @4000000055f79d3118998dd4 2015-09-15 04:23:03,412 [production-uswest2_kafka01.us-west-2.prod.conviva.com-1441843384564-a04e07b7-leader-finder-thread] ERROR FATAL kafka.consumer.ConsumerFetcherManager$LeaderFinderThread - [production-uswest2_kafka01.us-west-2.prod.conviva.com-1441843384564-a04e07b7-leader-finder-thread], Encountered NullPointerException. Exiting... @4000000055f79d3118a29a3c 2015-09-15 04:23:03,412 [ConsumerFetcherThread-production-uswest2_kafka01.us-west-2.prod.conviva.com-1441843384564-e723525e-0-15] WARN kafka.consumer.ConsumerFetcherThread - [ConsumerFetcherThread-production-uswest2_kafka01.us-west-2.prod.conviva.com-1441843384564-e723525e-0-15], Error in fetch Name: FetchRequest; Version: 0; CorrelationId: 1179876; ClientId: production-uswest2-ConsumerFetcherThread-production-uswest2_kafka01.us-west-2.prod.conviva.com-1441843384564-e723525e-0-15; ReplicaId: -1; MaxWait: 100 ms; MinBytes: 1 bytes; RequestInfo: [GOInfoTable,9] -> PartitionFetchInfo(454170,1000000) @4000000055f79d3118a2c14c java.nio.channels.UnresolvedAddressException: null @4000000055f79d3118a2c534 at sun.nio.ch.Net.checkAddress(Net.java:30) ~[na:1.6.0_26] @4000000055f79d3118a2c534 at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:480) ~[na:1.6.0_26] @4000000055f79d3118a2c91c at kafka.network.BlockingChannel.connect(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3118a2cd04 at kafka.consumer.SimpleConsumer.connect(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3118a2d0ec at kafka.consumer.SimpleConsumer.reconnect(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3118a2fbe4 at kafka.consumer.SimpleConsumer.liftedTree1$1(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3118a2ffcc at kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3118a303b4 at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3118a804f4 at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3118a804f4 at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3118a808dc at kafka.metrics.KafkaTimer.time(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3118a8187c at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3118a81c64 at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3118a81c64 at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3118a8204c at kafka.metrics.KafkaTimer.time(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3118a82fec at kafka.consumer.SimpleConsumer.fetch(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3118a82fec at kafka.server.AbstractFetcherThread.processFetchRequest(Unknown Source) [mirror-maker.jar:na] @4000000055f79d3118a833d4 at kafka.server.AbstractFetcherThread.doWork(Unknown Source) [mirror-maker.jar:na] @4000000055f79d3118a833d4 at kafka.utils.ShutdownableThread.run(Unknown Source) [mirror-maker.jar:na] @4000000055f79d3118a83ba4 2015-09-15 04:23:03,412 [production-uswest2_kafka01.us-west-2.prod.conviva.com-1441843384564-1a4f6a5b-leader-finder-thread] ERROR FATAL kafka.consumer.ConsumerFetcherManager$LeaderFinderThread - [production-uswest2_kafka01.us-west-2.prod.conviva.com-1441843384564-1a4f6a5b-leader-finder-thread], Encountered NullPointerException. Exiting... @4000000055f79d3118a83f8c 2015-09-15 04:23:03,413 [production-uswest2_kafka01.us-west-2.prod.conviva.com-1441843384564-b1e3f9ab-leader-finder-thread] ERROR FATAL kafka.consumer.ConsumerFetcherManager$LeaderFinderThread - [production-uswest2_kafka01.us-west-2.prod.conviva.com-1441843384564-b1e3f9ab-leader-finder-thread], Encountered NullPointerException. Exiting... @4000000055f79d3118a84f2c 2015-09-15 04:23:03,413 [ConsumerFetcherThread-production-uswest2_kafka01.us-west-2.prod.conviva.com-1441843384564-a04e07b7-0-14] WARN kafka.consumer.ConsumerFetcherThread - [ConsumerFetcherThread-production-uswest2_kafka01.us-west-2.prod.conviva.com-1441843384564-a04e07b7-0-14], Error in fetch Name: FetchRequest; Version: 0; CorrelationId: 1183391; ClientId: production-uswest2-ConsumerFetcherThread-production-uswest2_kafka01.us-west-2.prod.conviva.com-1441843384564-a04e07b7-0-14; ReplicaId: -1; MaxWait: 100 ms; MinBytes: 1 bytes; RequestInfo: [GOInfoTable,4] -> PartitionFetchInfo(453222,1000000) @4000000055f79d3118a85ecc java.nio.channels.UnresolvedAddressException: null @4000000055f79d3118a85ecc at sun.nio.ch.Net.checkAddress(Net.java:30) ~[na:1.6.0_26] @4000000055f79d3118a862b4 at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:480) ~[na:1.6.0_26] @4000000055f79d3118a862b4 at kafka.network.BlockingChannel.connect(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3118a8669c at kafka.consumer.SimpleConsumer.connect(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3118a8669c at kafka.consumer.SimpleConsumer.reconnect(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3118a86e6c at kafka.consumer.SimpleConsumer.liftedTree1$1(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3118a87254 at kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3118a87254 at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3118a889c4 at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3118a88dac at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3118a89194 at kafka.metrics.KafkaTimer.time(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3118a8957c at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3118a89964 at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3118a89964 at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3118a89d4c at kafka.metrics.KafkaTimer.time(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3118a8a904 at kafka.consumer.SimpleConsumer.fetch(Unknown Source) ~[mirror-maker.jar:na] @4000000055f79d3118a8a904 at kafka.server.AbstractFetcherThread.processFetchRequest(Unknown Source) [mirror-maker.jar:na] @4000000055f79d3118a8acec at kafka.server.AbstractFetcherThread.doWork(Unknown Source) [mirror-maker.jar:na]