[FLINK-22554] Support Kafka Topic Patterns in Kafka Ingress

• Type: Improvement
• Status: Open
• Priority: Not a Priority
• Resolution: Unresolved
• Affects Version/s: statefun-3.1.0
• Component/s: Stateful Functions
• Labels: auto-deprioritized-major auto-deprioritized-minor auto-unassigned pull-request-available
• Assignee: Unassigned
• Reporter: Seth Wiesman
• Created: 03/May/21 16:51
• Updated: 14/Jul/22 22:38

Description

Flink's Kafka source supports subscription patterns, where it will consume all topics that match a specified regex and periodically monitor for new topics. We should support this in the statefun Kafka ingress as it is generally useful and would remove a source of cluster downtime (subscribing to a new topic).

 

I propose something like this.

 

topics:
    - topic-pattern: my-topic-* // some regex
      discovery-interval: 10s  // some duration 
      valueType: blah
      targets: 
          - blah

The Flink consumer can be configured with both a list of concrete topics + a pattern so validation is simple.

 

cc igalshilman

Issue Links

• duplicates: FLINK-15821 Allow configuring Kafka partition / topic discovery in StateFun Kafka ingress
• links to: GitHub Pull Request #231