Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-13700

Kafka reporting CorruptRecordException exception

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Critical
    • Resolution: Unresolved
    • 2.4.1
    • None
    • core
    • None
    • ubuntu 16.04
      kafka 2.4

    Description

      In our kafka cluster a couple of partitions in __consumer_offsets and 1 regular topic getting data corruption issue while replicas trying to read from leader.  Similar messages for other partitions as well . 
       
      [2022-02-28 21:57:29,941] ERROR [ReplicaFetcher replicaId=6, leaderId=1, fetcherId=2] Found invalid messages during fetch for partition __consumer_offsets-10 offset 108845487 (kafka.server.ReplicaFetcherThread)

      org.apache.kafka.common.errors.CorruptRecordException: Record is corrupt (stored crc = 1524235439) in topic partition __consumer_offsets-10
       
      another topic partitions with same errors
      [2022-02-28 22:17:00,235] ERROR [ReplicaFetcher replicaId=6, leaderId=1, fetcherId=0] Found invalid messages during fetch for partition px-11351-xxxxxx-a56c642-0 offset 11746872 (kafka.server.ReplicaFetcherThread)
      org.apache.kafka.common.errors.CorruptRecordException: Record is corrupt (stored crc = 475179617) in topic partition px-11351-xxxxxx-a56c642-0.
       
      I have verified all infrastructure, dish network and system for any errors found and nothing found. I am not sure why it is happening or how to troubleshoot.  
       
      Bellow is output of the message from DumpLogSegments , 
       
      $ /opt/ns/kafka/bin/kafka-run-class.sh kafka.tools.DumpLogSegments --verify-index-only --deep-iteration --files ./00000000000011324034.log | grep 11746872
      baseOffset: 11746872 lastOffset: 11746872 count: 1 baseSequence: 50278 lastSequence: 50278 producerId: 17035 producerEpoch: 0 partitionLeaderEpoch: 8 isTransactional: false isControl: false position: 252530345 CreateTime: 1645886348240 size: 647 magic: 2 compresscodec: SNAPPY crc: 475179617 isvalid: true

      offset: 11746872 CreateTime: 1645886348240 keysize: 54 valuesize: 637 sequence: 50278 headerKeys: []

      Attachments

        Activity

          People

            Unassigned Unassigned
            ujalagam Uday Bhaskar
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: