Kafka
  1. Kafka
  2. KAFKA-544

Retain key in producer and expose it in the consumer

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:

      Description

      KAFKA-506 added support for retaining a key in the messages, however this field is not yet set by the producer.

      The proposal for doing this is to change the producer api to change ProducerData to allow only a single key/value pair so it has a one-to-one mapping to Message. That is change from
      ProducerData(topic: String, key: K, data: Seq[V])
      to
      ProducerData(topic: String, key: K, data: V)

      The key itself needs to be encoded. There are several ways this could be handled. A few of the options:
      1. Change the Encoder and Decoder to be MessageEncoder and MessageDecoder and have them take both a key and value.
      2. Another option is to change the type of the encoder/decoder to not refer to Message so it could be used for both the key and value.

      I favor the second option but am open to feedback.

      One concern with our current approach to serialization as well as both of these proposals is that they are inefficient. We go from Object=>byte[]=>Message=>MessageSet with a copy at each step. In the case of compression there are a bunch of intermediate steps. We could theoretically clean this up by instead having an interface for the encoder that was something like
      Encoder.writeTo(buffer: ByteBuffer, object: AnyRef)
      and
      Decoder.readFrom(buffer:ByteBuffer): AnyRef
      However there are two problems with this. The first is that we don't actually know the size of the data until it is serialized so we can't really allocate the bytebuffer properly and might need to resize it. The second is that in the case of compression there is a whole other path to consider. Originally I thought maybe it would be good to try to fix this, but now I think it should be out-of-scope and we should revisit the efficiency issue in a future release in conjunction with our internal handling of compression.

      1. KAFKA-544-v1.patch
        122 kB
        Jay Kreps
      2. KAFKA-544-v2.patch
        175 kB
        Jay Kreps
      3. KAFKA-544-v3.patch
        175 kB
        Jay Kreps
      4. KAFKA-544-v4.patch
        177 kB
        Jay Kreps
      5. KAFKA-544-v5.patch
        177 kB
        Jay Kreps
      6. KAFKA-544-v6.patch
        177 kB
        Jay Kreps
      7. KAFKA-544-joel-comments.patch
        8 kB
        Jay Kreps
      8. KAFKA-544-joel-comments-v2.patch
        10 kB
        Jay Kreps

        Activity

        Jay Kreps created issue -
        Jun Rao made changes -
        Field Original Value New Value
        Labels bugs
        Priority Major [ 3 ] Blocker [ 1 ]
        Jay Kreps made changes -
        Summary Retain key in producer Retain key in producer and expose it in the consumer
        Jay Kreps made changes -
        Attachment KAFKA-544-v1.patch [ 12553457 ]
        Jay Kreps made changes -
        Attachment KAFKA-544-v2.patch [ 12553569 ]
        Jay Kreps made changes -
        Attachment KAFKA-544-v3.patch [ 12553574 ]
        Jay Kreps made changes -
        Attachment KAFKA-544-v4.patch [ 12553600 ]
        Jay Kreps made changes -
        Attachment KAFKA-544-v5.patch [ 12553614 ]
        Jay Kreps made changes -
        Attachment KAFKA-544-v6.patch [ 12553699 ]
        Jay Kreps made changes -
        Attachment KAFKA-544-joel-comments.patch [ 12554550 ]
        Jay Kreps made changes -
        Attachment KAFKA-544-joel-comments-v2.patch [ 12554634 ]
        Jay Kreps made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Jun Rao made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Jay Kreps
            Reporter:
            Jay Kreps
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development