[FLINK-35187] FLIP-435: Introduce a New Materialized Table for Simplifying Data Pipelines

• Type: New Feature
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.20.0
• Fix Version/s: 2.0.0, 1.20.0
• Component/s: Table SQL / API, Table SQL / Gateway

Release Note

In 1.20, we introduced the Materialized Table in Flink SQL, a new table type designed to simplify both batch and stream data pipelines while providing a consistent development experience. By specifying data freshness and query at creation, the engine automatically derives the schema and creates a data refresh pipeline to maintain the specified freshness. More information about this feature can be found here: [Materialized Table Overview](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/materialized-table/overview/)

• Assignee: dalongliu
• Reporter: dalongliu
• Created: 22/Apr/24 07:39
• Updated: 10/Jul/24 07:04

Description

This is an umbrella issue for FLIP-435: Introduce a New Materialized Table for Simplifying Data Pipelines, see FLIP design doc for more detail: https://cwiki.apache.org/confluence/display/FLINK/FLIP-435%3A+Introduce+a+New+Materialized+Table+for+Simplifying+Data+Pipelines

Issue Links

• is a parent of: FLINK-35345 FLIP-448: Introduce Pluggable Workflow Scheduler Interface for Materialized Table
• mentioned in: Page Loading...
• mentioned in: Page Loading...

Sub-Tasks

• Introduce CatalogMaterializedTable and related interface to support materialized table: Resolved
• Introduce test-filesystem Catalog based on FileSystem Connector to support materialized table: Resolved
• Support create materialized table syntax: Resolved
• Support alter materialized table related syntaxes: suspend, resume, refresh, set and reset: Resolved
• Support drop materialized table syntax and execution in continuous refresh mode: Resolved
• Support describe job syntax and execution: Resolved
• Support the execution of create materialized table in continuous refresh mode: Resolved
• Support the execution of suspend, resume materialized table in continuous refresh mode: Resolved
• Support the execution of refresh materialized table: Resolved
• Support the execution of create materialized table in full refresh mode: Resolved
• Support the execution of suspend, resume materialized table in full refresh mode: Resolved
• Support the execution of drop materialized table in full refresh mode: Resolved
• Support the execution of alter materialized table set freshness: Open
• Support the execution of alter materialized table set refresh mode: Open
• Support the execution of alter materialized table set options: Open
• Support the execution of alter materialized table reset options: Open
• Desc table syntax support materialized table: Open
• Add doc for syntax `describe job 'xxx'`: Resolved
• Add end-to-end test for materialized table: Resolved
• Add materialized table statement doc: Resolved
• Add materialized table user guide doc: Open
• Add materialized table quickstart doc: Resolved
• Add gateway refresh rest api doc: Resolved
• Fix TestFileSystem Connector error setting execution mode: Resolved
• Add materialized table develop guide docs: Open
• Fix the bugs found in releasing testing of Materialized Table: Resolved
• Remove hugo_extended package introduced by wrong operation: Resolved
• Do not override the user-defined checkpoint interval in continuous mode.: Resolved
• Fix the incorrect partition generation during period refresh in Full Mode: Resolved