Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-8240

Create unified interfaces to configure and instatiate TableSources

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.5.0
    • Component/s: Table SQL / API
    • Labels:
      None

      Description

      At the moment every table source has different ways for configuration and instantiation. Some table source are tailored to a specific encoding (e.g., KafkaAvroTableSource, KafkaJsonTableSource) or only support one encoding for reading (e.g., CsvTableSource). Each of them might implement a builder or support table source converters for external catalogs.

      The table sources should have a unified interface for discovery, defining common properties, and instantiation. The TableSourceConverters provide a similar functionality but use an external catalog. We might generialize this interface.

      In general a table source declaration depends on the following parts:

      - Source
        - Type (e.g. Kafka, Custom)
        - Properties (e.g. topic, connection info)
      - Encoding
        - Type (e.g. Avro, JSON, CSV)
        - Schema (e.g. Avro class, JSON field names/types)
      - Rowtime descriptor/Proctime
        - Watermark strategy and Watermark properties
        - Time attribute info
      - Bucketization
      

      This issue needs a design document before implementation. Any discussion is very welcome.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                twalthr Timo Walther
                Reporter:
                twalthr Timo Walther
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: