[FLINK-6442] Extend TableAPI Support Sink Table Registration and ‘insert into’ Clause in SQL

• Type: New Feature
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 1.4.0
• Component/s: Table SQL / API
• Assignee: lincoln lee
• Reporter: lincoln lee
• Created: 04/May/17 09:22
• Updated: 08/Nov/17 14:41
• Resolved: 08/Nov/17 14:41

Description

Currently in TableAPI there’s only registration method for source table, when we use SQL writing a streaming job, we should add additional part for the sink, like TableAPI does:

val sqlQuery = "SELECT * FROM MyTable WHERE _1 = 3"

val t = StreamTestData.getSmall3TupleDataStream(env)
tEnv.registerDataStream("MyTable", t)

// one way: invoke tableAPI’s writeToSink method directly
val result = tEnv.sql(sqlQuery)
result.writeToSink(new YourStreamSink)

// another way: convert to datastream first and then invoke addSink 
val result = tEnv.sql(sqlQuery).toDataStream[Row]
result.addSink(new StreamITCase.StringSink)

From the api we can see the sink table always be a derived table because its 'schema' is inferred from the result type of upstream query.
Compare to traditional RDBMS which support DML syntax, a query with a target output could be written like this:

insert into table target_table_name
[(column_name [ ,...n ])]
query

The equivalent form of the example above is as follows:

    tEnv.registerTableSink("targetTable", new YourSink)

    val sql = "INSERT INTO targetTable SELECT a, b, c FROM sourceTable"
    val result = tEnv.sql(sql)

It is supported by Calcite’s grammar:

 insert:( INSERT | UPSERT ) INTO tablePrimary
 [ '(' column [, column ]* ')' ]
 query

I'd like to extend Flink TableAPI to support such feature. see design doc: https://goo.gl/n3phK5

Issue Links

• blocks: FLINK-7478 Update documentation for sql insert and api change in TableAPI & SQL
• links to: GitHub Pull Request #3829