Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-4412

Replication fetch stuck in loop on offset null

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: replication
    • Labels:

      Description

      I kicked off a cluster rebalance and it never completed. I had to look at node eth0 traffic to see there was a constant 60MB/s (I'm usually at about 5MB/s ingest). The /kafka/logs/server.log was looping like this, then I deleted the topic in question to make it stop:

      [2016-11-15 18:21:27,745] ERROR Found invalid messages during fetch for partition [cisco-2016.11.13,19] offset 861323 error null (kafka.server.ReplicaFetcherThread)
      [2016-11-15 18:21:27,755] ERROR Found invalid messages during fetch for partition [cisco-2016.11.13,19] offset 861323 error null (kafka.server.ReplicaFetcherThread)
      [2016-11-15 18:21:27,773] ERROR Found invalid messages during fetch for partition [cisco-2016.11.13,19] offset 861323 error null (kafka.server.ReplicaFetcherThread)
      [2016-11-15 18:21:27,788] ERROR Found invalid messages during fetch for partition [cisco-2016.11.13,19] offset 861323 error null (kafka.server.ReplicaFetcherThread)
      [2016-11-15 18:21:27,847] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions [cisco-2016.11.13,19] (kafka.server.ReplicaFetcherManager)
      [2016-11-15 18:21:27,852] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions [cisco-2016.11.13,11] (kafka.server.ReplicaFetcherManager)
      [2016-11-15 18:21:27,853] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions [cisco-2016.11.13,9] (kafka.server.ReplicaFetcherManager)
      [2016-11-15 18:21:27,855] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions [cisco-2016.11.13,3] (kafka.server.ReplicaFetcherManager)
      [2016-11-15 18:21:27,856] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions [cisco-2016.11.13,16] (kafka.server.ReplicaFetcherManager)
      [2016-11-15 18:21:27,857] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions [cisco-2016.11.13,2] (kafka.server.ReplicaFetcherManager)
      [2016-11-15 18:21:27,858] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions [cisco-2016.11.13,19] (kafka.server.ReplicaFetcherManager)
      [2016-11-15 18:21:28,012] INFO Deleting index /data/cisco-2016.11.13-19/00000000000000000000.index (kafka.log.OffsetIndex)
      [2016-11-15 18:21:28,016] INFO Deleted log for partition [cisco-2016.11.13,19] in /data/cisco-2016.11.13-19. (kafka.log.LogManager)
      [2016-11-15 18:21:28,024] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions [cisco-2016.11.13,11] (kafka.server.ReplicaFetcherManager)
      [2016-11-15 18:21:28,165] INFO Deleting index /data/cisco-2016.11.13-11/00000000000000000000.index (kafka.log.OffsetIndex)
      [2016-11-15 18:21:28,165] INFO Deleted log for partition [cisco-2016.11.13,11] in /data/cisco-2016.11.13-11. (kafka.log.LogManager)
      [2016-11-15 18:21:28,167] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions [cisco-2016.11.13,9] (kafka.server.ReplicaFetcherManager)
      [2016-11-15 18:21:28,232] INFO Deleting index /data/cisco-2016.11.13-9/00000000000000000000.index (kafka.log.OffsetIndex)
      [2016-11-15 18:21:28,232] INFO Deleted log for partition [cisco-2016.11.13,9] in /data/cisco-2016.11.13-9. (kafka.log.LogManager)
      [2016-11-15 18:21:28,242] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions [cisco-2016.11.13,3] (kafka.server.ReplicaFetcherManager)
      [2016-11-15 18:21:28,341] INFO Deleting index /data/cisco-2016.11.13-3/00000000000000000000.index (kafka.log.OffsetIndex)
      [2016-11-15 18:21:28,342] INFO Deleted log for partition [cisco-2016.11.13,3] in /data/cisco-2016.11.13-3. (kafka.log.LogManager)
      [2016-11-15 18:21:28,375] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions [cisco-2016.11.13,16] (kafka.server.ReplicaFetcherManager)
      [2016-11-15 18:21:28,465] INFO Deleting index /data/cisco-2016.11.13-16/00000000000000000000.index (kafka.log.OffsetIndex)
      [2016-11-15 18:21:28,466] INFO Deleted log for partition [cisco-2016.11.13,16] in /data/cisco-2016.11.13-16. (kafka.log.LogManager)
      [2016-11-15 18:21:28,469] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions [cisco-2016.11.13,2] (kafka.server.ReplicaFetcherManager)
      [2016-11-15 18:21:28,486] INFO Deleting index /data/cisco-2016.11.13-2/00000000000000000000.index (kafka.log.OffsetIndex)
      [2016-11-15 18:21:28,486] INFO Deleted log for partition [cisco-2016.11.13,2] in /data/cisco-2016.11.13-2. (kafka.log.LogManager)

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              xrl Xavier Lange
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: