Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.10.0.0
    • Fix Version/s: 0.10.0.0
    • Component/s: streams
    • Labels:
      None

      Description

      The only way for Processor implementations to control partitioning of forwarded messages is to set the partitioner class as property ProducerConfig.PARTITIONER_CLASS_CONFIG in the StreamsConfig, which should be set to the name of a org.apache.kafka.clients.producer.Partitioner implementation. However, doing this requires the partitioner knowing how to properly partition all topics, not just the one or few topics used by the Processor.

      Instead, Kafka Streams should make it easy to optionally add a partitioning function for each sink used in a topology. Each sink represents a single output topic, and thus is far simpler to implement. Additionally, the sink is already typed with the key and value types (via serdes for the keys and values), so the partitioner can be also be typed with the key and value types. Finally, this also keeps the logic of choosing partitioning strategies where it belongs, as part of building the topology.

        Attachments

          Activity

            People

            • Assignee:
              rhauch Randall Hauch
              Reporter:
              rhauch Randall Hauch
              Reviewer:
              Guozhang Wang
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: