[FLINK-11199] Support interactive programming in Flink

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.0.0
• Component/s: Table SQL / API
• Assignee: Jiangjie Qin
• Reporter: Jiangjie Qin
• Created: 19/Dec/18 08:19
• Updated: 06/Nov/20 14:32
• Resolved: 06/Nov/20 14:32

Description

Generally speaking, Flink applications may consist of one or more jobs, and the jobs may want to share the data with others. In Flink, the jobs in the same application are independent and share nothing among themselves. If a Flink application involves several sequential steps, each step (as an independent job) will have to write its intermediate results to an external sink, so that its results can be used by the following step (job) as sources. Although this works, that means users have to provide external storage and explicitly write/read from it. 

 

To improve the user experience, we would like to propose adding support of interactive programming in Flink. We would like to start with Table API. And the discussion thread is following:

https://docs.google.com/document/d/1d4T2zTyfe7hdncEUAxrlNOYr4e5IMNEZLyqSuuswkA0/edit?ts=5bf639d9#

Issue Links

• duplicates: FLINK-19343 FLIP-36: Support Interactive Programming in Flink
• duplicates: FLINK-10867 Add a DataSet-based CacheOperator to reuse results between jobs

Sub-Tasks

• Introduce BLOCKING_PERSISTENT result partition type: Closed
• Report BLOCKING_PERSISTENT result partition meta back to client: Closed
• Add support for consuming BLOCKING_PERSISTENT ResultPartition: Closed
• Add support of cache/invalidateCache for TableAPI: Closed
• Implement RPCs to allow clients release result partitions in a Flink cluster.: Closed