[FLINK-18449] Make topic discovery and partition discovery configurable for FlinkKafkaConsumer in Table API

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.11.0
• Fix Version/s: 1.12.0
• Component/s: Connectors / Kafka, Table SQL / API
• Labels: pull-request-available
• Assignee: Shengkai Fang
• Reporter: Shengkai Fang
• Created: 29/Jun/20 12:52
• Updated: 30/Apr/21 09:21
• Resolved: 20/Aug/20 02:55

Description

In streaming api, we can use regex to find topic and enable partiton discovery by setting non-negative value for `flink.partition-discovery.interval-millis`. However, it's not work in table api. We propose the following new connector options:

'topic' = 'topic'                                // for single topic
'topic' = 'topic-1, topic-2,..., topic-n',       // for list topic
'topic-pattern' = 'topic*'                       // for use of regex,  "topic" and "topic-pattern" can't coexist
'scan.topic-partition-discovery.interval' = '5s'  // for both topic discovery

Issue Links

• Blocked: FLINK-18602 Support specific offset for topic list for kafka connector
• duplicates: FLINK-14729 Multi-topics consuming from KafkaTableSource
• duplicates: FLINK-13340 Add more Kafka topic option of flink-connector-kafka
• relates to: FLINK-13052 Supporting multi-topic when using kafkaTableSourceSinkFactoryBase.createStreamTableSource
• links to: GitHub Pull Request #12908