[FLINK-3221] Move Table API on top of Apache Calcite

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Implemented
• Fix Version/s: 1.1.0
• Component/s: Table SQL / API
• Labels: requires-design-doc
• Assignee: Unassigned
• Reporter: Fabian Hueske
• Created: 12/Jan/16 17:01
• Updated: 27/Apr/21 18:04
• Resolved: 18/Mar/16 14:02

Description

The goal of this task is to move the Table API on top of Calcite and preserve the feature set of the batch part. Calcite is a SQL optimization and translation framework. It includes a SQL Parser, a catalog component, a validator, and a rule-based (relational) optimizer. This is the first step for adding SQL and StreamSQL support to Flink.

Since the streaming part of the Table API is rather a proof-of-concept implementation and only features selection and projection, we exclude it from this task. With the current Table API, queries can only be embedded into DataSet (or DataStream) programs with manually defined schema for DataSet/DataStreams. We do not add support for external data stores (CSV, Parquet, ORC files, HBase, etc.) as part of this task. Instead we only consume data that comes from a DataSet/DataStream operator.

More details can be found in the design document for SQL / StreamSQL on Flink.

Issue Links

• Blocked: FLINK-2099 Add a SQL API

Sub-Tasks

• Translate Table API query into logical relational plans for Apache Calcite: Resolved
• Optimize logical Table API plans in Calcite: Resolved
• Translate optimized logical Table API plans into physical plans representing DataSet programs: Resolved
• Translate physical Table API plan into DataSet program: Closed
• Implement Calc Support: Resolved
• Implement Union Support: Resolved
• Refactor Table API before merging into master: Resolved
• Fix/adjust Table API examples: Resolved