Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-8190

Add extra FlinkKafkaConsumer constructors to expose pattern-based topic subscription functionality

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.4.0, 1.5.0
    • Component/s: Connectors / Kafka
    • Labels:
      None

      Description

      The required internals for pattern-based topic discovery was implemented as part of FLINK-4022 (along with partition discovery). However, the functionality for pattern-based topic discovery was not yet exposed via any visible user API on the version-specific subclasses of FlinkKafkaConsumerBase.

      I propose to add two more constructors for this:

      public FlinkKafkaConsumerXX(java.util.regex.Pattern subscriptionPattern, DeserializationSchema<T> schema, Properties props);
      
      public FlinkKafkaConsumerXX(java.util.regex.Pattern subscriptionPattern, KeyedDeserializationSchema<T> schema, Properties props);
      

      This allows the consumer to pick up all matching topics on startup.
      To continuously pick up matching topics on the fly when they are created after the job has already started running, users should additionally set the KEY_PARTITION_DISCOVERY_INTERVAL_MILLIS property, as they would do for partition discovery.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                tzulitai Tzu-Li (Gordon) Tai
                Reporter:
                tzulitai Tzu-Li (Gordon) Tai
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: