Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-6687

Allow to read a topic multiple times

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.8.0
    • streams
    • None

    Description

      Users often want to read topic multiple times. However, this is not possible because there is a single consumer and thus a topic can only be consumed once.

      Users get an exception

      Exception in thread “main” org.apache.kafka.streams.errors.TopologyException: Invalid topology: Topic source has already been registered by another source.

      If they use a topic name in multiple `stream()`, `table()`, `globalTable()` calls.

      However, with KAFKA-6034 in place, we could allow adding a topic multiple times and rewrite the topology internally to only read the topic once. This would simplify application code as users don't need to put workaround in place to get the same behavior.

      Attachments

        Issue Links

          is related to

          Improvement - An improvement or enhancement to an existing feature or task. KAFKA-6686 Allow reading topic as KStream and GlobalKTable

          • Major - Major loss of function.
          • Open
          Parent Feature

          Improvement - An improvement or enhancement to an existing feature or task. KAFKA-6034 Streams DSL to Processor Topology Translation Improvements

          • Major - Major loss of function.
          • Open
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. KAFKA-10721 Rewrite topology to allow for overlapping unequal topic subscriptions

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • In Progress
          links to

          Web Link GitHub Pull Request #9582

          Web Link GitHub Pull Request #9609

          Activity

            People

              ableegoldman A. Sophie Blee-Goldman
              mjsax Matthias J. Sax
              Votes:
              8 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: