[FLINK-15985] offload runtime params from DDL to table hints in DML/queries

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Component/s: Table SQL / API
• Assignee: Danny Chen
• Reporter: Bowen Li
• Created: 11/Feb/20 04:40
• Updated: 26/Apr/20 10:10
• Resolved: 26/Apr/20 10:10

Description

background:

Currently Flink DDL mixes three types of params all together: 

• External data’s metadata: defines what the data looks like (schema), where it is (location/url), how it should be accessed (username/pwd)
• Source/sink runtime params: defines how and usually how fast Flink source/sink reads/writes data, not affecting the results
• Kafka “sink-partitioner”
• Elastic “bulk-flush.interval/max-size/...”

• Semantics params: defines aspects like how much data Flink reads/writes, how the result will look like
• Kafka “startup-mode”, “offset”
• Watermark, timestamp column

 

Problems of the current mix-up: Flink cannot leverage catalogs and external system metadata alone to run queries with all the non-metadata params involved in DDL. E.g. when we add a catalog for Confluent Schema Registry, the expected user experience should be that Flink users just configure the catalog with url and usr/pwd, and should be able to run queries immediately; however, that’s not the case right now because users still have to use DDL to define a bunch params like “startup-mode”, “offset”, timestamp column, etc, along with the schema redundantly. We’ve heard many user complaints on this.

 

cc ykt836 lirui lzljs3620320 jark twalthr dwysakowicz

Issue Links

• is duplicated by: FLINK-17101 [Umbrella] Supports dynamic table options for Flink SQL