Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-13602

Allow to broadcast a result record

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 3.4.0
    • streams

    Description

      From time to time, users ask how they can send a record to more than one partition in a sink topic. Currently, this is only possible by replicate the message N times before the sink and use a custom partitioner to write the N messages into the N different partitions.

      It might be worth to make this easier and add a new feature for it. There are multiple options:

      • extend `to()` / `addSink()` with a "broadcast" option/config
      • add `toAllPartitions()` / `addBroadcastSink()` methods
      • allow StreamPartitioner to return `-1` for "all partitions"
      • extend `StreamPartitioner` to allow returning more than one partition (ie a list/collection of integers instead of a single int)

      The first three options imply that a "full broadcast" is supported only, so it's less flexible. On the other hand, it's easier to use (especially the first two options are easy as they do not require to implement a custom partitioner).

      The last option would be most flexible and also allow for a "partial broadcast" (aka multi-cast) pattern. It might also be possible to combine two options, or maye even a totally different idea.

      Attachments

        Activity

          People

            sagarrao Sagar Rao
            mjsax Matthias J. Sax
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: