Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-2590 KIP-28: Kafka Streams Checklist
  3. KAFKA-2649

Add support for custom partitioner in sink nodes

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 0.10.0.0
    • 0.10.0.0
    • streams
    • None

    Description

      The only way for Processor implementations to control partitioning of forwarded messages is to set the partitioner class as property ProducerConfig.PARTITIONER_CLASS_CONFIG in the StreamsConfig, which should be set to the name of a org.apache.kafka.clients.producer.Partitioner implementation. However, doing this requires the partitioner knowing how to properly partition all topics, not just the one or few topics used by the Processor.

      Instead, Kafka Streams should make it easy to optionally add a partitioning function for each sink used in a topology. Each sink represents a single output topic, and thus is far simpler to implement. Additionally, the sink is already typed with the key and value types (via serdes for the keys and values), so the partitioner can be also be typed with the key and value types. Finally, this also keeps the logic of choosing partitioning strategies where it belongs, as part of building the topology.

      Attachments

        Activity

          People

            rhauch Randall Hauch
            rhauch Randall Hauch
            Guozhang Wang Guozhang Wang
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: