Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-14644 Improve the usability of connector properties
  3. FLINK-14649

Flatten all the connector properties keys to make it easy to configure in DDL

    XMLWordPrintableJSON

    Details

    • Release Note:
      Hide
      Some indexed connector properties have been refactored to have a better experience in DDL statement, including Kafka "connector.properties", "connector.specific-offsets" and Elasticsearch "connector.hosts". Please see documentation for the description of new properties. The old properties are deprecated and will be removed in future versions. You can find the new properties here: https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/connect.htm
      Show
      Some indexed connector properties have been refactored to have a better experience in DDL statement, including Kafka "connector.properties", "connector.specific-offsets" and Elasticsearch "connector.hosts". Please see documentation for the description of new properties. The old properties are deprecated and will be removed in future versions. You can find the new properties here: https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/connect.htm

      Description

      There are some connector properties are lists. For example, Kafka connector specific properties have to been set in the following way:

       'connector.properties.0.key' = 'zookeeper.connect',
        'connector.properties.0.value' = 'localhost:2181',
        'connector.properties.1.key' = 'bootstrap.servers',
        'connector.properties.1.value' = 'localhost:9092',
        'connector.properties.2.key' = 'group.id',
        'connector.properties.2.value' = 'testGroup',
      

      It is complex and not intuitive to define in this way. In order to cooperate with DDL better, we propose to flatten all the property keys.

      It has some disadvantage to define in this way.

      • Users need to keep track of the indices
      • The key space is not constant. Validation of keys would require prefix magic and wildcards. Like in TableFactories: `connector.propertie.#.key`.
      • It is complex and not intuitive to define and document.

      See FLIP-86 for the proposed new properties.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Leonard Xu Leonard Xu
                Reporter:
                jark Jark Wu
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 20m
                  20m