Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-727

KafkaIO should support dynamic addition of Kafka partitions to assigned topics.

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: P3
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: io-java-kafka
    • Labels:
      None

      Description

      Kafka topics may add partitions dynamically (doesn't require Kafka to restart, or halt a topic), and the KafkaIO should probably support this.

      Note:
      Consistently assigning partitions should be taken into account, specifically for the case of reading from multiple topics, where one (or more) of the topics added partitions while the pipeline is running.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                amitsela Amit Sela
              • Votes:
                1 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated: