Kafka
  1. Kafka
  2. KAFKA-925

Add optional partition key override in producer

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.1
    • Fix Version/s: 0.8.1
    • Component/s: producer
    • Labels:
      None

      Description

      We have a key that is used for partitioning in the producer and stored with the message. Actually these uses, though often the same, could be different. The two meanings are effectively:
      1. Assignment to a partition
      2. Deduplication within a partition

      In cases where we want to allow the client to take advantage of both of these and they aren't the same it would be nice to allow them to be specified separately.

      To implement this I added an optional partition key to KeyedMessage. When specified this key is used for partitioning rather than the message key. This key is of type Any and the parametric typing is removed from the partitioner to allow it to work with either key.

      An alternative would be to allow the partition id to specified in the KeyedMessage. This would be slightly more convenient in the case where there is no partition key but instead you know a priori the partition number-this case must be handled by giving the partition id as the partition key and using an identity partitioner which is slightly more roundabout. However this is inconsistent with the normal partitioning which requires a key in the case where the partition is determined by a key-in that case you would be manually calling your partitioner in user code. It seems best to me to either use a key or always a partition and since we currently take a key I stuck with that.

      1. KAFKA-925-post-commit-v1.patch
        3 kB
        Jay Kreps
      2. KAFKA-925-v1.patch
        16 kB
        Jay Kreps
      3. KAFKA-925-v2.patch
        18 kB
        Jay Kreps

        Activity

        Jay Kreps created issue -
        Jay Kreps made changes -
        Field Original Value New Value
        Attachment KAFKA-925-v1.patch [ 12585228 ]
        Jay Kreps made changes -
        Attachment KAFKA-925-v2.patch [ 12592856 ]
        Jay Kreps made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Fix Version/s 0.8.1 [ 12322960 ]
        Resolution Fixed [ 1 ]
        Jay Kreps made changes -
        Attachment KAFKA-925-post-commit-v1.patch [ 12599841 ]

          People

          • Assignee:
            Jay Kreps
            Reporter:
            Jay Kreps
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development