[FLINK-14649] Flatten all the connector properties keys to make it easy to configure in DDL

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.10.0
• Component/s: Table SQL / API
• Labels: pull-request-available

Release Note

Some indexed connector properties have been refactored to have a better experience in DDL statement, including Kafka "connector.properties", "connector.specific-offsets" and Elasticsearch "connector.hosts". Please see documentation for the description of new properties. The old properties are deprecated and will be removed in future versions. You can find the new properties here: https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/connect.htm

• Assignee: Leonard Xu
• Reporter: Jark Wu
• Created: 07/Nov/19 07:00
• Updated: 18/Sep/22 15:12
• Resolved: 09/Dec/19 12:19

Description

There are some connector properties are lists. For example, Kafka connector specific properties have to been set in the following way:

 'connector.properties.0.key' = 'zookeeper.connect',
  'connector.properties.0.value' = 'localhost:2181',
  'connector.properties.1.key' = 'bootstrap.servers',
  'connector.properties.1.value' = 'localhost:9092',
  'connector.properties.2.key' = 'group.id',
  'connector.properties.2.value' = 'testGroup',

It is complex and not intuitive to define in this way. In order to cooperate with DDL better, we propose to flatten all the property keys.

It has some disadvantage to define in this way.

• Users need to keep track of the indices
• The key space is not constant. Validation of keys would require prefix magic and wildcards. Like in TableFactories: `connector.propertie.#.key`.
• It is complex and not intuitive to define and document.

See FLIP-86 for the proposed new properties.

Issue Links

• links to: GitHub Pull Request #10468