Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-22554

Support Kafka Topic Patterns in Kafka Ingress

Attach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

      Description

      Flink's Kafka source supports subscription patterns, where it will consume all topics that match a specified regex and periodically monitor for new topics. We should support this in the statefun Kafka ingress as it is generally useful and would remove a source of cluster downtime (subscribing to a new topic).

       

      I propose something like this.

       

      topics:
          - topic-pattern: my-topic-* // some regex
            discovery-interval: 10s  // some duration 
            valueType: blah
            targets: 
                - blah

      The Flink consumer can be configured with both a list of concrete topics + a pattern so validation is simple.

       

      cc Igal Shilman

        Attachments

        Issue Links

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              sjwiesman Seth Wiesman

              Dates

              • Created:
                Updated:

                Issue deployment