[2017-04-24 22:46:58,146] ERROR [ReplicaFetcherThread-0-73], Error for partition [__consumer_offsets,30] to broker 73:org.apache.kafka.c ommon.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread) [2017-04-24 22:46:58,148] INFO [ReplicaFetcherManager on broker 74] Added fetcher for partitions List([__consumer_offsets-39, initOffset 0 to broker BrokerEndPoint(73,10.52.208.73,9092)] ) (kafka.server.ReplicaFetcherManager) [2017-04-24 22:46:58,148] ERROR [ReplicaFetcherThread-0-73], Error for partition [__consumer_offsets,39] to broker 73:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread) [2017-04-24 22:46:58,150] INFO [ReplicaFetcherManager on broker 74] Removed fetcher for partitions __consumer_offsets-18 (kafka.server.ReplicaFetcherManager) [2017-04-24 22:46:58,150] INFO Truncating log __consumer_offsets-18 to offset 0. (kafka.log.Log) [2017-04-24 22:46:58,152] INFO [ReplicaFetcherManager on broker 74] Added fetcher for partitions List([__consumer_offsets-18, initOffset 0 to broker BrokerEndPoint(73,10.52.208.73,9092)] ) (kafka.server.ReplicaFetcherManager) [2017-04-24 22:46:58,153] ERROR [ReplicaFetcherThread-0-73], Error for partition [__consumer_offsets,18] to broker 73:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread) [2017-04-24 22:46:58,154] INFO [ReplicaFetcherManager on broker 74] Removed fetcher for partitions __consumer_offsets-0 (kafka.server.ReplicaFetcherManager) [2017-04-24 22:46:58,154] INFO Truncating log __consumer_offsets-0 to offset 0. (kafka.log.Log) [2017-04-24 22:46:58,156] INFO [ReplicaFetcherManager on broker 74] Added fetcher for partitions List([__consumer_offsets-0, initOffset 0 to broker BrokerEndPoint(73,10.52.208.73,9092)] ) (kafka.server.ReplicaFetcherManager) [2017-04-24 22:46:58,157] ERROR [ReplicaFetcherThread-0-73], Error for partition [__consumer_offsets,0] to broker 73:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread) [2017-04-24 22:46:58,158] INFO [ReplicaFetcherManager on broker 74] Removed fetcher for partitions __consumer_offsets-24 (kafka.server.ReplicaFetcherManager) [2017-04-24 22:46:58,159] INFO Truncating log __consumer_offsets-24 to offset 0. (kafka.log.Log) [2017-04-24 22:46:58,161] INFO [ReplicaFetcherManager on broker 74] Added fetcher for partitions List([__consumer_offsets-24, initOffset 0 to broker BrokerEndPoint(73,10.52.208.73,9092)] ) (kafka.server.ReplicaFetcherManager) [2017-04-24 22:46:58,161] ERROR [ReplicaFetcherThread-0-73], Error for partition [__consumer_offsets,24] to broker 73:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread) [2017-04-24 22:46:58,163] INFO [ReplicaFetcherManager on broker 74] Removed fetcher for partitions IL2S-0 (kafka.server.ReplicaFetcherManager) [2017-04-24 22:46:58,163] INFO Truncating log IL2S-0 to offset 2859157863. (kafka.log.Log) [2017-04-24 22:46:58,180] WARN [ReplicaFetcherThread-0-72], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@9251219 (kafka.server.ReplicaFetcherThread) java.io.IOException: Connection to 72 was disconnected before the response was read at kafka.utils.NetworkClientBlockingOps$.$anonfun$blockingSendAndReceive$3(NetworkClientBlockingOps.scala:114) at kafka.utils.NetworkClientBlockingOps$.$anonfun$blockingSendAndReceive$3$adapted(NetworkClientBlockingOps.scala:112) at scala.Option.foreach(Option.scala:257) at kafka.utils.NetworkClientBlockingOps$.$anonfun$blockingSendAndReceive$1(NetworkClientBlockingOps.scala:112) at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:136) at kafka.utils.NetworkClientBlockingOps$.pollContinuously$extension(NetworkClientBlockingOps.scala:142) at kafka.utils.NetworkClientBlockingOps$.blockingSendAndReceive$extension(NetworkClientBlockingOps.scala:108) at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:249) at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:234) at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42) at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:118) at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:103) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63) [2017-04-24 22:46:58,224] INFO [ReplicaFetcherManager on broker 74] Added fetcher for partitions List([IL2S-0, initOffset 2859157863 to broker BrokerEndPoint(73,10.52.208.73,9092)] ) (kafka.server.ReplicaFetcherManager) [2017-04-24 22:46:58,226] INFO [ReplicaFetcherManager on broker 74] Removed fetcher for partitions __consumer_offsets-33 (kafka.server.ReplicaFetcherManager) [2017-04-24 22:46:58,227] INFO Truncating log __consumer_offsets-33 to offset 0. (kafka.log.Log) [2017-04-24 22:46:58,229] INFO [ReplicaFetcherManager on broker 74] Added fetcher for partitions List([__consumer_offsets-33, initOffset 0 to broker BrokerEndPoint(73,10.52.208.73,9092)] ) (kafka.server.ReplicaFetcherManager) [2017-04-24 22:46:58,231] INFO [ReplicaFetcherManager on broker 74] Removed fetcher for partitions test-2 (kafka.server.ReplicaFetcherManager) [2017-04-24 22:46:58,231] INFO Truncating log test-2 to offset 5. (kafka.log.Log) [2017-04-24 22:46:58,233] INFO [ReplicaFetcherManager on broker 74] Added fetcher for partitions List([test-2, initOffset 5 to broker BrokerEndPoint(73,10.52.208.73,9092)] ) (kafka.server.ReplicaFetcherManager) [2017-04-24 22:46:58,235] INFO [ReplicaFetcherManager on broker 74] Removed fetcher for partitions __consumer_offsets-3 (kafka.server.ReplicaFetcherManager) [2017-04-24 22:46:58,235] INFO Truncating log __consumer_offsets-3 to offset 0. (kafka.log.Log) [2017-04-24 22:46:58,237] INFO [ReplicaFetcherManager on broker 74] Added fetcher for partitions List([__consumer_offsets-3, initOffset 0 to broker BrokerEndPoint(73,10.52.208.73,9092)] ) (kafka.server.ReplicaFetcherManager) [2017-04-24 22:46:58,239] INFO [ReplicaFetcherManager on broker 74] Removed fetcher for partitions __consumer_offsets-21 (kafka.server.ReplicaFetcherManager) [2017-04-24 22:46:58,239] INFO Truncating log __consumer_offsets-21 to offset 0. (kafka.log.Log) [2017-04-24 22:46:58,242] INFO [ReplicaFetcherManager on broker 74] Added fetcher for partitions List([__consumer_offsets-21, initOffset 0 to broker BrokerEndPoint(73,10.52.208.73,9092)] ) (kafka.server.ReplicaFetcherManager) [2017-04-24 22:46:58,244] INFO [ReplicaFetcherManager on broker 74] Removed fetcher for partitions SEGMENT-0 (kafka.server.ReplicaFetcherManager) [2017-04-24 22:46:58,244] INFO Truncating log SEGMENT-0 to offset 101667030. (kafka.log.Log) [2017-04-24 22:46:58,247] INFO [ReplicaFetcherManager on broker 74] Added fetcher for partitions List([SEGMENT-0, initOffset 101667030 to broker BrokerEndPoint(73,10.52.208.73,9092)] ) (kafka.server.ReplicaFetcherManager) [2017-04-24 22:46:58,249] INFO [ReplicaFetcherManager on broker 74] Removed fetcher for partitions __consumer_offsets-12 (kafka.server.ReplicaFetcherManager) [2017-04-24 22:46:58,250] INFO Truncating log __consumer_offsets-12 to offset 0. (kafka.log.Log) [2017-04-24 22:46:58,252] INFO [ReplicaFetcherManager on broker 74] Added fetcher for partitions List([__consumer_offsets-12, initOffset 0 to broker BrokerEndPoint(73,10.52.208.73,9092)] ) (kafka.server.ReplicaFetcherManager) [2017-04-24 22:46:58,255] INFO [ReplicaFetcherManager on broker 74] Removed fetcher for partitions __consumer_offsets-15 (kafka.server.ReplicaFetcherManager) [2017-04-24 22:46:58,255] INFO Truncating log __consumer_offsets-15 to offset 0. (kafka.log.Log) [2017-04-24 22:46:58,258] INFO [ReplicaFetcherManager on broker 74] Added fetcher for partitions List([__consumer_offsets-15, initOffset 0 to broker BrokerEndPoint(73,10.52.208.73,9092)] ) (kafka.server.ReplicaFetcherManager) [2017-04-24 22:46:58,260] INFO [ReplicaFetcherManager on broker 74] Removed fetcher for partitions __consumer_offsets-48 (kafka.server.ReplicaFetcherManager) [2017-04-24 22:46:58,261] INFO Truncating log __consumer_offsets-48 to offset 0. (kafka.log.Log) [2017-04-24 22:46:58,262] INFO [ReplicaFetcherManager on broker 74] Added fetcher for partitions List([__consumer_offsets-48, initOffset 0 to broker BrokerEndPoint(73,10.52.208.73,9092)] ) (kafka.server.ReplicaFetcherManager) [2017-04-24 22:46:58,264] INFO [ReplicaFetcherManager on broker 74] Removed fetcher for partitions __consumer_offsets-6 (kafka.server.ReplicaFetcherManager) [2017-04-24 22:46:58,265] INFO Truncating log __consumer_offsets-6 to offset 0. (kafka.log.Log) [2017-04-24 22:46:58,267] INFO [ReplicaFetcherManager on broker 74] Added fetcher for partitions List([__consumer_offsets-6, initOffset 0 to broker BrokerEndPoint(73,10.52.208.73,9092)] ) (kafka.server.ReplicaFetcherManager) [2017-04-24 22:46:58,269] INFO [ReplicaFetcherManager on broker 74] Removed fetcher for partitions __consumer_offsets-42 (kafka.server.ReplicaFetcherManager) [2017-04-24 22:46:58,269] INFO Truncating log __consumer_offsets-42 to offset 0. (kafka.log.Log) [2017-04-24 22:46:58,271] INFO [ReplicaFetcherManager on broker 74] Added fetcher for partitions List([__consumer_offsets-42, initOffset 0 to broker BrokerEndPoint(73,10.52.208.73,9092)] ) (kafka.server.ReplicaFetcherManager) [2017-04-24 22:46:58,274] INFO [ReplicaFetcherManager on broker 74] Removed fetcher for partitions __consumer_offsets-27 (kafka.server.ReplicaFetcherManager) [2017-04-24 22:46:58,274] INFO Truncating log __consumer_offsets-27 to offset 0. (kafka.log.Log) [2017-04-24 22:46:58,277] INFO [ReplicaFetcherManager on broker 74] Added fetcher for partitions List([__consumer_offsets-27, initOffset 0 to broker BrokerEndPoint(73,10.52.208.73,9092)] ) (kafka.server.ReplicaFetcherManager) [2017-04-24 22:46:58,281] INFO [ReplicaFetcherManager on broker 74] Removed fetcher for partitions __consumer_offsets-45 (kafka.server.ReplicaFetcherManager) [2017-04-24 22:46:58,281] INFO Truncating log __consumer_offsets-45 to offset 0. (kafka.log.Log) [2017-04-24 22:46:58,284] INFO [ReplicaFetcherManager on broker 74] Added fetcher for partitions List([__consumer_offsets-45, initOffset 0 to broker BrokerEndPoint(73,10.52.208.73,9092)] ) (kafka.server.ReplicaFetcherManager) [2017-04-24 22:46:58,287] INFO [ReplicaFetcherManager on broker 74] Removed fetcher for partitions DSLAM-1 (kafka.server.ReplicaFetcherManager) [2017-04-24 22:46:58,287] INFO Truncating log DSLAM-1 to offset 3097004023. (kafka.log.Log) [2017-04-24 22:46:58,294] INFO [ReplicaFetcherManager on broker 74] Added fetcher for partitions List([DSLAM-1, initOffset 3097004023 to broker BrokerEndPoint(73,10.52.208.73,9092)] ) (kafka.server.ReplicaFetcherManager) [2017-04-24 22:46:58,296] INFO [ReplicaFetcherManager on broker 74] Removed fetcher for partitions __consumer_offsets-36 (kafka.server.ReplicaFetcherManager) [2017-04-24 22:46:58,296] INFO Truncating log __consumer_offsets-36 to offset 0. (kafka.log.Log) [2017-04-24 22:46:58,298] INFO [ReplicaFetcherManager on broker 74] Added fetcher for partitions List([__consumer_offsets-36, initOffset 0 to broker BrokerEndPoint(73,10.52.208.73,9092)] ) (kafka.server.ReplicaFetcherManager) [2017-04-24 22:46:58,300] INFO [ReplicaFetcherManager on broker 74] Removed fetcher for partitions __consumer_offsets-9 (kafka.server.ReplicaFetcherManager) [2017-04-24 22:46:58,300] INFO Truncating log __consumer_offsets-9 to offset 0. (kafka.log.Log) [2017-04-24 22:46:58,303] INFO [ReplicaFetcherManager on broker 74] Added fetcher for partitions List([__consumer_offsets-9, initOffset 0 to broker BrokerEndPoint(73,10.52.208.73,9092)] ) (kafka.server.ReplicaFetcherManager) [2017-04-24 22:46:58,394] ERROR [ReplicaFetcherThread-0-73], Error for partition [IL2S,0] to broker 73:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread) [2017-04-24 22:46:58,401] ERROR [ReplicaFetcherThread-0-73], Error for partition [__consumer_offsets,21] to broker 73:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread) [2017-04-24 22:46:58,401] ERROR [ReplicaFetcherThread-0-73], Error for partition [__consumer_offsets,27] to broker 73:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread) [2017-04-24 22:46:58,401] ERROR [ReplicaFetcherThread-0-73], Error for partition [__consumer_offsets,9] to broker 73:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread) [2017-04-24 22:46:58,401] ERROR [ReplicaFetcherThread-0-73], Error for partition [__consumer_offsets,9] to broker 73:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread) [2017-04-24 22:46:58,401] ERROR [ReplicaFetcherThread-0-73], Error for partition [__consumer_offsets,33] to broker 73:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread) [2017-04-24 22:46:58,401] ERROR [ReplicaFetcherThread-0-73], Error for partition [__consumer_offsets,36] to broker 73:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread) [2017-04-24 22:46:58,401] ERROR [ReplicaFetcherThread-0-73], Error for partition [__consumer_offsets,42] to broker 73:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread) [2017-04-24 22:46:58,401] ERROR [ReplicaFetcherThread-0-73], Error for partition [__consumer_offsets,3] to broker 73:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread) [2017-04-24 22:46:58,401] ERROR [ReplicaFetcherThread-0-73], Error for partition [__consumer_offsets,15] to broker 73:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread) [2017-04-24 22:46:58,401] ERROR [ReplicaFetcherThread-0-73], Error for partition [__consumer_offsets,48] to broker 73:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread) [2017-04-24 22:46:58,401] ERROR [ReplicaFetcherThread-0-73], Error for partition [__consumer_offsets,6] to broker 73:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread) [2017-04-24 22:46:58,401] ERROR [ReplicaFetcherThread-0-73], Error for partition [__consumer_offsets,12] to broker 73:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread) [2017-04-24 22:46:58,401] ERROR [ReplicaFetcherThread-0-73], Error for partition [__consumer_offsets,45] to broker 73:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread) [2017-04-24 22:46:58,402] ERROR [ReplicaFetcherThread-0-73], Error for partition [DSLAM,1] to broker 73:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread) [2017-04-24 22:46:58,402] ERROR [ReplicaFetcherThread-0-73], Error for partition [SEGMENT,0] to broker 73:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread) [2017-04-24 22:46:58,402] ERROR [ReplicaFetcherThread-0-73], Error for partition [test,2] to broker 73:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread) [2017-04-24 22:48:08,495] INFO [Group Metadata Manager on Broker 74]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager) [2017-04-24 22:53:01,330] INFO Rolled new log segment for 'SEGMENT-1' in 2 ms. (kafka.log.Log) [2017-04-24 22:53:04,026] INFO Rolled new log segment for 'SEGMENT-0' in 1 ms. (kafka.log.Log) [2017-04-24 22:53:06,609] INFO Rolled new log segment for 'SEGMENT-2' in 1 ms. (kafka.log.Log) [2017-04-24 22:58:08,495] INFO [Group Metadata Manager on Broker 74]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager) [2017-04-24 23:02:37,929] INFO Rolled new log segment for 'DSLAM-1' in 1 ms. (kafka.log.Log) [2017-04-24 23:08:08,495] INFO [Group Metadata Manager on Broker 74]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager) [2017-04-24 23:15:51,979] INFO Rolled new log segment for 'DSLAM-0' in 1 ms. (kafka.log.Log) [2017-04-24 23:16:08,131] INFO Rolled new log segment for 'DSLAM-2' in 1 ms. (kafka.log.Log) [2017-04-24 23:18:08,495] INFO [Group Metadata Manager on Broker 74]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager) [2017-04-24 23:18:38,293] INFO Scheduling log segment 2350710897 for log IL2S-0 for deletion. (kafka.log.Log) [2017-04-24 23:19:38,294] INFO Deleting segment 2350710897 from log IL2S-0. (kafka.log.Log) [2017-04-24 23:19:38,314] INFO Deleting index /npm/application/Telemetry/kafka_2.12-0.10.2.0/logs/IL2S-0/00000000002350710897.index.deleted (kafka.log.OffsetIndex) [2017-04-24 23:19:38,316] INFO Deleting index /npm/application/Telemetry/kafka_2.12-0.10.2.0/logs/IL2S-0/00000000002350710897.timeindex.deleted (kafka.log.TimeIndex) [2017-04-24 23:23:38,293] INFO Scheduling log segment 2591917648 for log DSLAM-1 for deletion. (kafka.log.Log) [2017-04-24 23:23:38,294] INFO Scheduling log segment 2592043489 for log DSLAM-0 for deletion. (kafka.log.Log) [2017-04-24 23:23:38,295] INFO Scheduling log segment 2592030929 for log DSLAM-2 for deletion. (kafka.log.Log) [2017-04-24 23:24:38,294] INFO Deleting segment 2591917648 from log DSLAM-1. (kafka.log.Log) [2017-04-24 23:24:38,295] INFO Deleting segment 2592043489 from log DSLAM-0. (kafka.log.Log) [2017-04-24 23:24:38,295] INFO Deleting segment 2592030929 from log DSLAM-2. (kafka.log.Log) [2017-04-24 23:24:38,308] INFO Deleting index /npm/application/Telemetry/kafka_2.12-0.10.2.0/logs/DSLAM-2/00000000002592030929.index.deleted (kafka.log.OffsetIndex) [2017-04-24 23:24:38,308] INFO Deleting index /npm/application/Telemetry/kafka_2.12-0.10.2.0/logs/DSLAM-0/00000000002592043489.index.deleted (kafka.log.OffsetIndex) [2017-04-24 23:24:38,308] INFO Deleting index /npm/application/Telemetry/kafka_2.12-0.10.2.0/logs/DSLAM-1/00000000002591917648.index.deleted (kafka.log.OffsetIndex) [2017-04-24 23:24:38,309] INFO Deleting index /npm/application/Telemetry/kafka_2.12-0.10.2.0/logs/DSLAM-2/00000000002592030929.timeindex.deleted (kafka.log.TimeIndex) [2017-04-24 23:24:38,309] INFO Deleting index /npm/application/Telemetry/kafka_2.12-0.10.2.0/logs/DSLAM-1/00000000002591917648.timeindex.deleted (kafka.log.TimeIndex) [2017-04-24 23:24:38,309] INFO Deleting index /npm/application/Telemetry/kafka_2.12-0.10.2.0/logs/DSLAM-0/00000000002592043489.timeindex.deleted (kafka.log.TimeIndex) [2017-04-24 23:28:08,495] INFO [Group Metadata Manager on Broker 74]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager) [2017-04-24 23:38:08,495] INFO [Group Metadata Manager on Broker 74]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager) [2017-04-24 23:45:14,006] INFO Creating /controller (is it secure? false) (kafka.utils.ZKCheckedEphemeral) [2017-04-24 23:45:14,009] INFO Result of znode creation is: NODEEXISTS (kafka.utils.ZKCheckedEphemeral) [2017-04-24 23:45:14,013] INFO New leader is 73 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener) [2017-04-24 23:45:14,433] INFO [ReplicaFetcherManager on broker 74] Removed fetcher for partitions DSLAM-0,__consumer_offsets-35,__consumer_offsets-41,__consumer_offsets-23,__consumer_offsets-47,__consumer_offsets-17,__consumer_offsets-11,__consumer_offsets-5,__consumer_offsets-29 (kafka.server.ReplicaFetcherManager) [2017-04-24 23:45:14,439] INFO [ReplicaFetcherManager on broker 74] Removed fetcher for partitions __consumer_offsets-26,__consumer_offsets-32,__consumer_offsets-8,IL2S-2,__consumer_offsets-38,__consumer_offsets-14,__consumer_offsets-44,__consumer_offsets-20,SEGMENT-2,test-1,__consumer_offsets-2 (kafka.server.ReplicaFetcherManager) [2017-04-24 23:45:14,440] INFO Truncating log IL2S-2 to offset 2836802018. (kafka.log.Log) [2017-04-24 23:45:14,440] INFO Truncating log __consumer_offsets-8 to offset 0. (kafka.log.Log) [2017-04-24 23:45:14,440] INFO Truncating log __consumer_offsets-38 to offset 0. (kafka.log.Log) [2017-04-24 23:45:14,440] INFO Truncating log test-1 to offset 6. (kafka.log.Log) [2017-04-24 23:45:14,440] INFO Truncating log __consumer_offsets-2 to offset 0. (kafka.log.Log) [2017-04-24 23:45:14,440] INFO Truncating log __consumer_offsets-14 to offset 0. (kafka.log.Log) [2017-04-24 23:45:14,440] INFO Truncating log SEGMENT-2 to offset 102363057. (kafka.log.Log) [2017-04-24 23:45:14,441] INFO Truncating log __consumer_offsets-20 to offset 0. (kafka.log.Log) [2017-04-24 23:45:14,442] INFO Truncating log __consumer_offsets-44 to offset 0. (kafka.log.Log) [2017-04-24 23:45:14,442] INFO Truncating log __consumer_offsets-26 to offset 2. (kafka.log.Log) [2017-04-24 23:45:14,442] INFO Truncating log __consumer_offsets-32 to offset 2. (kafka.log.Log) [2017-04-24 23:45:14,446] INFO [ReplicaFetcherManager on broker 74] Added fetcher for partitions List([IL2S-2, initOffset 2836802018 to broker BrokerEndPoint(73,10.52.208.73,9092)] , [__consumer_offsets-8, initOffset 0 to broker BrokerEndPoint(73,10.52.208.73,9092)] , [__consumer_offsets-38, initOffset 0 to broker BrokerEndPoint(73,10.52.208.73,9092)] , [test-1, initOffset 6 to broker BrokerEndPoint(73,10.52.208.73,9092)] , [__consumer_offsets-2, initOffset 0 to broker BrokerEndPoint(73,10.52.208.73,9092)] , [__consumer_offsets-14, initOffset 0 to broker BrokerEndPoint(73,10.52.208.73,9092)] , [SEGMENT-2, initOffset 102363057 to broker BrokerEndPoint(73,10.52.208.73,9092)] , [__consumer_offsets-20, initOffset 0 to broker BrokerEndPoint(73,10.52.208.73,9092)] , [__consumer_offsets-44, initOffset 0 to broker BrokerEndPoint(73,10.52.208.73,9092)] , [__consumer_offsets-26, initOffset 2 to broker BrokerEndPoint(73,10.52.208.73,9092)] , [__consumer_offsets-32, initOffset 2 to broker BrokerEndPoint(73,10.52.208.73,9092)] ) (kafka.server.ReplicaFetcherManager) [2017-04-24 23:45:14,447] INFO [ReplicaFetcherThread-0-72], Shutting down (kafka.server.ReplicaFetcherThread) [2017-04-24 23:45:18,371] INFO Partition [DSLAM,2] on broker 74: Shrinking ISR for partition [DSLAM,2] from 72,73,74 to 73,74 (kafka.cluster.Partition) [2017-04-24 23:45:18,594] INFO Partition [__consumer_offsets,31] on broker 74: Shrinking ISR for partition [__consumer_offsets,31] from 72,73,74 to 73,74 (kafka.cluster.Partition) [2017-04-24 23:45:23,371] INFO Partition [__consumer_offsets,13] on broker 74: Shrinking ISR for partition [__consumer_offsets,13] from 72,73,74 to 73,74 (kafka.cluster.Partition) [2017-04-24 23:45:23,374] INFO Partition [SEGMENT,1] on broker 74: Shrinking ISR for partition [SEGMENT,1] from 72,73,74 to 73,74 (kafka.cluster.Partition) [2017-04-24 23:45:23,377] INFO Partition [__consumer_offsets,46] on broker 74: Shrinking ISR for partition [__consumer_offsets,46] from 72,73,74 to 73,74 (kafka.cluster.Partition) [2017-04-24 23:45:23,379] INFO Partition [IL2S,1] on broker 74: Shrinking ISR for partition [IL2S,1] from 72,73,74 to 73,74 (kafka.cluster.Partition) [2017-04-24 23:45:23,381] INFO Partition [__consumer_offsets,1] on broker 74: Shrinking ISR for partition [__consumer_offsets,1] from 72,73,74 to 73,74 (kafka.cluster.Partition) [2017-04-24 23:45:23,383] INFO Partition [__consumer_offsets,34] on broker 74: Shrinking ISR for partition [__consumer_offsets,34] from 72,73,74 to 73,74 (kafka.cluster.Partition) [2017-04-24 23:45:23,385] INFO Partition [__consumer_offsets,16] on broker 74: Shrinking ISR for partition [__consumer_offsets,16] from 72,73,74 to 73,74 (kafka.cluster.Partition) [2017-04-24 23:45:23,387] INFO Partition [__consumer_offsets,49] on broker 74: Shrinking ISR for partition [__consumer_offsets,49] from 72,73,74 to 73,74 (kafka.cluster.Partition) [2017-04-24 23:45:23,389] INFO Partition [__consumer_offsets,25] on broker 74: Shrinking ISR for partition [__consumer_offsets,25] from 72,73,74 to 73,74 (kafka.cluster.Partition) [2017-04-24 23:45:23,391] INFO Partition [__consumer_offsets,37] on broker 74: Shrinking ISR for partition [__consumer_offsets,37] from 72,73,74 to 73,74 (kafka.cluster.Partition) [2017-04-24 23:45:23,392] INFO Partition [__consumer_offsets,4] on broker 74: Shrinking ISR for partition [__consumer_offsets,4] from 72,73,74 to 73,74 (kafka.cluster.Partition) [2017-04-24 23:45:23,394] INFO Partition [__consumer_offsets,19] on broker 74: Shrinking ISR for partition [__consumer_offsets,19] from 72,73,74 to 73,74 (kafka.cluster.Partition) [2017-04-24 23:45:23,396] INFO Partition [__consumer_offsets,28] on broker 74: Shrinking ISR for partition [__consumer_offsets,28] from 72,73,74 to 73,74 (kafka.cluster.Partition) [2017-04-24 23:45:23,397] INFO Partition [__consumer_offsets,7] on broker 74: Shrinking ISR for partition [__consumer_offsets,7] from 72,73,74 to 73,74 (kafka.cluster.Partition) [2017-04-24 23:45:23,399] INFO Partition [__consumer_offsets,40] on broker 74: Shrinking ISR for partition [__consumer_offsets,40] from 72,73,74 to 73,74 (kafka.cluster.Partition) [2017-04-24 23:45:23,401] INFO Partition [__consumer_offsets,43] on broker 74: Shrinking ISR for partition [__consumer_offsets,43] from 72,73,74 to 73,74 (kafka.cluster.Partition) [2017-04-24 23:45:23,403] INFO Partition [test,0] on broker 74: Shrinking ISR for partition [test,0] from 72,73,74 to 73,74 (kafka.cluster.Partition) [2017-04-24 23:45:23,404] INFO Partition [__consumer_offsets,10] on broker 74: Shrinking ISR for partition [__consumer_offsets,10] from 72,73,74 to 73,74 (kafka.cluster.Partition) [2017-04-24 23:45:23,406] INFO Partition [__consumer_offsets,22] on broker 74: Shrinking ISR for partition [__consumer_offsets,22] from 72,73,74 to 73,74 (kafka.cluster.Partition) [2017-04-24 23:45:38,445] INFO [ReplicaFetcherThread-0-72], Stopped (kafka.server.ReplicaFetcherThread) [2017-04-24 23:45:38,445] INFO [ReplicaFetcherThread-0-72], Shutdown completed (kafka.server.ReplicaFetcherThread) [2017-04-24 23:45:38,450] INFO [Group Metadata Manager on Broker 74]: Loading offsets and group metadata from __consumer_offsets-41 (kafka.coordinator.GroupMetadataManager) [2017-04-24 23:45:38,698] INFO [Group Metadata Manager on Broker 74]: Finished loading offsets from __consumer_offsets-41 in 247 milliseconds. (kafka.coordinator.GroupMetadataManager) [2017-04-24 23:45:38,698] INFO [Group Metadata Manager on Broker 74]: Loading offsets and group metadata from __consumer_offsets-47 (kafka.coordinator.GroupMetadataManager) [2017-04-24 23:45:38,700] INFO [Group Metadata Manager on Broker 74]: Finished loading offsets from __consumer_offsets-47 in 2 milliseconds. (kafka.coordinator.GroupMetadataManager) [2017-04-24 23:45:38,700] INFO [Group Metadata Manager on Broker 74]: Loading offsets and group metadata from __consumer_offsets-5 (kafka.coordinator.GroupMetadataManager) [2017-04-24 23:45:38,701] INFO [Group Metadata Manager on Broker 74]: Finished loading offsets from __consumer_offsets-5 in 1 milliseconds. (kafka.coordinator.GroupMetadataManager) [2017-04-24 23:45:38,701] INFO [Group Metadata Manager on Broker 74]: Loading offsets and group metadata from __consumer_offsets-11 (kafka.coordinator.GroupMetadataManager) [2017-04-24 23:45:38,703] INFO [Group Metadata Manager on Broker 74]: Finished loading offsets from __consumer_offsets-11 in 2 milliseconds. (kafka.coordinator.GroupMetadataManager) [2017-04-24 23:45:38,703] INFO [Group Metadata Manager on Broker 74]: Loading offsets and group metadata from __consumer_offsets-17 (kafka.coordinator.GroupMetadataManager) [2017-04-24 23:45:38,704] INFO [Group Metadata Manager on Broker 74]: Finished loading offsets from __consumer_offsets-17 in 1 milliseconds. (kafka.coordinator.GroupMetadataManager) [2017-04-24 23:45:38,704] INFO [Group Metadata Manager on Broker 74]: Loading offsets and group metadata from __consumer_offsets-23 (kafka.coordinator.GroupMetadataManager) [2017-04-24 23:45:38,706] INFO [Group Metadata Manager on Broker 74]: Finished loading offsets from __consumer_offsets-23 in 2 milliseconds. (kafka.coordinator.GroupMetadataManager) [2017-04-24 23:45:38,706] INFO [Group Metadata Manager on Broker 74]: Loading offsets and group metadata from __consumer_offsets-29 (kafka.coordinator.GroupMetadataManager) [2017-04-24 23:45:38,707] INFO [Group Metadata Manager on Broker 74]: Finished loading offsets from __consumer_offsets-29 in 1 milliseconds. (kafka.coordinator.GroupMetadataManager) [2017-04-24 23:45:38,707] INFO [Group Metadata Manager on Broker 74]: Loading offsets and group metadata from __consumer_offsets-35 (kafka.coordinator.GroupMetadataManager) [2017-04-24 23:45:38,709] INFO [Group Metadata Manager on Broker 74]: Finished loading offsets from __consumer_offsets-35 in 2 milliseconds. (kafka.coordinator.GroupMetadataManager) [2017-04-24 23:46:05,138] INFO Partition [__consumer_offsets,35] on broker 74: Expanding ISR for partition __consumer_offsets-35 from 73,74 to 73,74,72 (kafka.cluster.Partition) [2017-04-24 23:46:05,141] INFO Partition [__consumer_offsets,41] on broker 74: Expanding ISR for partition __consumer_offsets-41 from 73,74 to 73,74,72 (kafka.cluster.Partition) [2017-04-24 23:46:05,143] INFO Partition [__consumer_offsets,23] on broker 74: Expanding ISR for partition __consumer_offsets-23 from 73,74 to 73,74,72 (kafka.cluster.Partition) [2017-04-24 23:46:05,145] INFO Partition [__consumer_offsets,47] on broker 74: Expanding ISR for partition __consumer_offsets-47 from 73,74 to 73,74,72 (kafka.cluster.Partition) [2017-04-24 23:46:05,147] INFO Partition [__consumer_offsets,17] on broker 74: Expanding ISR for partition __consumer_offsets-17 from 73,74 to 73,74,72 (kafka.cluster.Partition) [2017-04-24 23:46:05,149] INFO Partition [__consumer_offsets,11] on broker 74: Expanding ISR for partition __consumer_offsets-11 from 73,74 to 73,74,72 (kafka.cluster.Partition) [2017-04-24 23:46:05,150] INFO Partition [__consumer_offsets,5] on broker 74: Expanding ISR for partition __consumer_offsets-5 from 73,74 to 73,74,72 (kafka.cluster.Partition) [2017-04-24 23:46:05,153] INFO Partition [__consumer_offsets,29] on broker 74: Expanding ISR for partition __consumer_offsets-29 from 73,74 to 73,74,72 (kafka.cluster.Partition) [2017-04-24 23:46:05,167] INFO Partition [DSLAM,0] on broker 74: Expanding ISR for partition DSLAM-0 from 73,74 to 73,74,72 (kafka.cluster.Partition) [2017-04-24 23:46:06,278] INFO Partition [SEGMENT,1] on broker 74: Expanding ISR for partition SEGMENT-1 from 73,74 to 73,74,72 (kafka.cluster.Partition) [2017-04-24 23:46:06,280] INFO Partition [__consumer_offsets,28] on broker 74: Expanding ISR for partition __consumer_offsets-28 from 73,74 to 73,74,72 (kafka.cluster.Partition) [2017-04-24 23:46:06,282] INFO Partition [__consumer_offsets,43] on broker 74: Expanding ISR for partition __consumer_offsets-43 from 73,74 to 73,74,72 (kafka.cluster.Partition) [2017-04-24 23:46:06,283] INFO Partition [__consumer_offsets,10] on broker 74: Expanding ISR for partition __consumer_offsets-10 from 73,74 to 73,74,72 (kafka.cluster.Partition) [2017-04-24 23:46:06,285] INFO Partition [__consumer_offsets,25] on broker 74: Expanding ISR for partition __consumer_offsets-25 from 73,74 to 73,74,72 (kafka.cluster.Partition) [2017-04-24 23:46:06,287] INFO Partition [__consumer_offsets,7] on broker 74: Expanding ISR for partition __consumer_offsets-7 from 73,74 to 73,74,72 (kafka.cluster.Partition) [2017-04-24 23:46:06,288] INFO Partition [__consumer_offsets,40] on broker 74: Expanding ISR for partition __consumer_offsets-40 from 73,74 to 73,74,72 (kafka.cluster.Partition) [2017-04-24 23:46:06,290] INFO Partition [__consumer_offsets,37] on broker 74: Expanding ISR for partition __consumer_offsets-37 from 73,74 to 73,74,72 (kafka.cluster.Partition) [2017-04-24 23:46:06,291] INFO Partition [__consumer_offsets,22] on broker 74: Expanding ISR for partition __consumer_offsets-22 from 73,74 to 73,74,72 (kafka.cluster.Partition) [2017-04-24 23:46:06,293] INFO Partition [test,0] on broker 74: Expanding ISR for partition test-0 from 73,74 to 73,74,72 (kafka.cluster.Partition) [2017-04-24 23:46:06,296] INFO Partition [__consumer_offsets,4] on broker 74: Expanding ISR for partition __consumer_offsets-4 from 73,74 to 73,74,72 (kafka.cluster.Partition) [2017-04-24 23:46:06,298] INFO Partition [__consumer_offsets,34] on broker 74: Expanding ISR for partition __consumer_offsets-34 from 73,74 to 73,74,72 (kafka.cluster.Partition) [2017-04-24 23:46:06,300] INFO Partition [__consumer_offsets,19] on broker 74: Expanding ISR for partition __consumer_offsets-19 from 73,74 to 73,74,72 (kafka.cluster.Partition) [2017-04-24 23:46:06,301] INFO Partition [__consumer_offsets,16] on broker 74: Expanding ISR for partition __consumer_offsets-16 from 73,74 to 73,74,72 (kafka.cluster.Partition) [2017-04-24 23:46:06,302] INFO Partition [__consumer_offsets,49] on broker 74: Expanding ISR for partition __consumer_offsets-49 from 73,74 to 73,74,72 (kafka.cluster.Partition) [2017-04-24 23:46:06,304] INFO Partition [__consumer_offsets,1] on broker 74: Expanding ISR for partition __consumer_offsets-1 from 73,74 to 73,74,72 (kafka.cluster.Partition) [2017-04-24 23:46:06,306] INFO Partition [__consumer_offsets,31] on broker 74: Expanding ISR for partition __consumer_offsets-31 from 73,74 to 73,74,72 (kafka.cluster.Partition) [2017-04-24 23:46:06,375] INFO Partition [__consumer_offsets,13] on broker 74: Expanding ISR for partition __consumer_offsets-13 from 73,74 to 73,74,72 (kafka.cluster.Partition) [2017-04-24 23:46:06,379] INFO Partition [__consumer_offsets,46] on broker 74: Expanding ISR for partition __consumer_offsets-46 from 73,74 to 73,74,72 (kafka.cluster.Partition) [2017-04-24 23:46:06,902] INFO Partition [DSLAM,2] on broker 74: Expanding ISR for partition DSLAM-2 from 73,74 to 73,74,72 (kafka.cluster.Partition) [2017-04-24 23:46:18,208] INFO Partition [IL2S,1] on broker 74: Expanding ISR for partition IL2S-1 from 73,74 to 73,74,72 (kafka.cluster.Partition) [2017-04-24 23:48:08,495] INFO [Group Metadata Manager on Broker 74]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager) [2017-04-24 23:50:12,283] WARN [ReplicaFetcherThread-0-73], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@5de6ee5a (kafka.server.ReplicaFetcherThread) java.io.IOException: Connection to 73 was disconnected before the response was read at kafka.utils.NetworkClientBlockingOps$.$anonfun$blockingSendAndReceive$3(NetworkClientBlockingOps.scala:114) at kafka.utils.NetworkClientBlockingOps$.$anonfun$blockingSendAndReceive$3$adapted(NetworkClientBlockingOps.scala:112) at scala.Option.foreach(Option.scala:257) at kafka.utils.NetworkClientBlockingOps$.$anonfun$blockingSendAndReceive$1(NetworkClientBlockingOps.scala:112) at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:136) at kafka.utils.NetworkClientBlockingOps$.pollContinuously$extension(NetworkClientBlockingOps.scala:142) at kafka.utils.NetworkClientBlockingOps$.blockingSendAndReceive$extension(NetworkClientBlockingOps.scala:108) at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:249) at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:234) at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42) at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:118) at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:103) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63) [2017-04-24 23:50:12,285] INFO Partition [__consumer_offsets,13] on broker 74: Shrinking ISR for partition [__consumer_offsets,13] from 73,74,72 to 74 (kafka.cluster.Partition) [2017-04-24 23:50:12,314] WARN Attempting to send response via channel for which there is no open connection, connection id 0 (kafka.network.Processor) [2017-04-24 23:50:12,314] WARN Attempting to send response via channel for which there is no open connection, connection id 4 (kafka.network.Processor) [2017-04-24 23:50:12,320] WARN Attempting to send response via channel for which there is no open connection, connection id 0 (kafka.network.Processor) [2017-04-24 23:50:12,332] WARN Attempting to send response via channel for which there is no open connection, connection id 2 (kafka.network.Processor) [2017-04-24 23:50:12,333] WARN Attempting to send response via channel for which there is no open connection, connection id 1 (kafka.network.Processor) [2017-04-24 23:50:12,333] WARN Attempting to send response via channel for which there is no open connection, connection id 1 (kafka.network.Processor) [2017-04-24 23:50:12,333] WARN Attempting to send response via channel for which there is no open connection, connection id 1 (kafka.network.Processor) [2017-04-24 23:50:12,339] WARN Attempting to send response via channel for which there is no open connection, connection id 1 (kafka.network.Processor) [2017-04-24 23:50:12,365] WARN Attempting to send response via channel for which there is no open connection, connection id 3 (kafka.network.Processor) [2017-04-24 23:50:12,472] WARN Attempting to send response via channel for which there is no open connection, connection id 3 (kafka.network.Processor) [2017-04-24 23:50:13,187] INFO re-registering broker info in ZK for broker 74 (kafka.server.KafkaHealthcheck$SessionExpireListener) [2017-04-24 23:50:13,187] INFO Creating /brokers/ids/74 (is it secure? false) (kafka.utils.ZKCheckedEphemeral) [2017-04-24 23:50:13,189] INFO Partition [__consumer_offsets,13] on broker 74: Cached zkVersion [65] not equal to that in zookeeper, skip updating ISR (kafka.cluster.Partition) [2017-04-24 23:50:13,189] INFO Partition [SEGMENT,1] on broker 74: Shrinking ISR for partition [SEGMENT,1] from 73,74,72 to 74 (kafka.cluster.Partition) [2017-04-24 23:50:13,190] INFO Result of znode creation is: OK (kafka.utils.ZKCheckedEphemeral) [2017-04-24 23:50:13,190] INFO Registered broker 74 at path /brokers/ids/74 with addresses: EndPoint(10.52.208.74,9092,ListenerName(PLAINTEXT),PLAINTEXT) (kafka.utils.ZkUtils) [2017-04-24 23:50:13,190] INFO done re-registering broker (kafka.server.KafkaHealthcheck$SessionExpireListener) [2017-04-24 23:50:13,190] INFO Subscribing to /brokers/topics path to watch for new topics (kafka.server.KafkaHealthcheck$SessionExpireListener) [2017-04-24 23:50:13,193] INFO New leader is 72 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener) [2017-04-24 23:50:13,194] INFO Partition [SEGMENT,1] on broker 74: Cached zkVersion [65] not equal to that in zookeeper, skip updating ISR (kafka.cluster.Partition) [2017-04-24 23:50:13,194] INFO Partition [__consumer_offsets,46] on broker 74: Shrinking ISR for partition [__consumer_offsets,46] from 73,74,72 to 74 (kafka.cluster.Partition) [2017-04-24 23:50:13,198] INFO Partition [__consumer_offsets,46] on broker 74: Cached zkVersion [63] not equal to that in zookeeper, skip updating ISR (kafka.cluster.Partition) [2017-04-24 23:50:13,199] INFO Partition [IL2S,1] on broker 74: Shrinking ISR for partition [IL2S,1] from 73,74,72 to 74 (kafka.cluster.Partition) [2017-04-24 23:50:13,202] INFO Partition [IL2S,1] on broker 74: Cached zkVersion [64] not equal to that in zookeeper, skip updating ISR (kafka.cluster.Partition) [2017-04-24 23:50:13,203] INFO Partition [__consumer_offsets,17] on broker 74: Shrinking ISR for partition [__consumer_offsets,17] from