Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-4815 Idempotent/transactional Producer (KIP-98)
  3. KAFKA-4935

Consider disabling record level CRC checks for message format V2

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • None
    • 0.11.0.0
    • None

    Description

      With the new message format proposed in KIP-98, the record level CRC has been moved to the the batch header.

      Because we expose the record-level CRC in `RecordMetadata` and `ConsumerRecord`, we currently compute it eagerly based on the key, value and timestamp even though these methods are rarely used. Ideally, we'd deprecate the relevant methods in `RecordMetadata` and `ConsumerRecord` while making the CRC computation lazy. This seems pretty hard to achieve in the Producer without increasing memory retention, but it may be possible to do in the Consumer.

      An alternative option is to return the batch CRC from the relevant methods.

      Attachments

        Issue Links

          is depended upon by

          Sub-task - The sub-task of the issue KAFKA-4816 Message format changes for idempotent/transactional producer

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link GitHub Pull Request #3123

          Activity

            People

              hachikuji Jason Gustafson
              apurva Apurva Mehta
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: