Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-512

Remove checksum from ByteBufferMessageSet.iterator

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.0
    • Component/s: None
    • Labels:

      Description

      Messages are explicitly checksumed in Log.append. But there is also a checksum computed and checked automatically in ByteBufferMessageSet.iterator as we iterate. This iterator is used quite a lot and as a result we compute this checksum 39 times on a single message produce. It turns out the default crc32 implementation in java is quite expensive so this is not good.

      The proposed fix is to remove the automatic checksum from the iterator and add explicit isValid() checks in the consumer as well as retaining the existing check in Log.append().

      If folks are in agreement I will probably include this in the KAFKA-506 patch as that already contains a lot of ByteBufferMessageSet changes.

        Attachments

          Activity

            People

            • Assignee:
              jkreps Jay Kreps
              Reporter:
              jkreps Jay Kreps
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: