Details
-
Improvement
-
Status: Open
-
Not a Priority
-
Resolution: Unresolved
-
statefun-3.1.0
-
None
Description
Flink's Kafka source supports subscription patterns, where it will consume all topics that match a specified regex and periodically monitor for new topics. We should support this in the statefun Kafka ingress as it is generally useful and would remove a source of cluster downtime (subscribing to a new topic).
I propose something like this.
topics: - topic-pattern: my-topic-* // some regex discovery-interval: 10s // some duration valueType: blah targets: - blah
The Flink consumer can be configured with both a list of concrete topics + a pattern so validation is simple.
cc igalshilman
Attachments
Issue Links
- duplicates
-
FLINK-15821 Allow configuring Kafka partition / topic discovery in StateFun Kafka ingress
- Closed
- links to