Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-14644 Improve the usability of connector properties
  3. FLINK-14649

Flatten all the connector properties keys to make it easy to configure in DDL

    XMLWordPrintableJSON

Details

    • Hide
      Some indexed connector properties have been refactored to have a better experience in DDL statement, including Kafka "connector.properties", "connector.specific-offsets" and Elasticsearch "connector.hosts". Please see documentation for the description of new properties. The old properties are deprecated and will be removed in future versions. You can find the new properties here: https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/connect.htm
      Show
      Some indexed connector properties have been refactored to have a better experience in DDL statement, including Kafka "connector.properties", "connector.specific-offsets" and Elasticsearch "connector.hosts". Please see documentation for the description of new properties. The old properties are deprecated and will be removed in future versions. You can find the new properties here: https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/connect.htm

    Description

      There are some connector properties are lists. For example, Kafka connector specific properties have to been set in the following way:

       'connector.properties.0.key' = 'zookeeper.connect',
        'connector.properties.0.value' = 'localhost:2181',
        'connector.properties.1.key' = 'bootstrap.servers',
        'connector.properties.1.value' = 'localhost:9092',
        'connector.properties.2.key' = 'group.id',
        'connector.properties.2.value' = 'testGroup',
      

      It is complex and not intuitive to define in this way. In order to cooperate with DDL better, we propose to flatten all the property keys.

      It has some disadvantage to define in this way.

      • Users need to keep track of the indices
      • The key space is not constant. Validation of keys would require prefix magic and wildcards. Like in TableFactories: `connector.propertie.#.key`.
      • It is complex and not intuitive to define and document.

      See FLIP-86 for the proposed new properties.

      Attachments

        Issue Links

          Activity

            People

              leonard Leonard Xu
              jark Jark Wu
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 20m
                  20m