Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-35187

FLIP-435: Introduce a New Materialized Table for Simplifying Data Pipelines

    XMLWordPrintableJSON

Details

    • Hide
      In 1.20, we introduced the Materialized Table in Flink SQL, a new table type designed to simplify both batch and stream data pipelines while providing a consistent development experience. By specifying data freshness and query at creation, the engine automatically derives the schema and creates a data refresh pipeline to maintain the specified freshness. More information about this feature can be found here: [Materialized Table Overview](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/materialized-table/overview/)
      Show
      In 1.20, we introduced the Materialized Table in Flink SQL, a new table type designed to simplify both batch and stream data pipelines while providing a consistent development experience. By specifying data freshness and query at creation, the engine automatically derives the schema and creates a data refresh pipeline to maintain the specified freshness. More information about this feature can be found here: [Materialized Table Overview]( https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/materialized-table/overview/ )

    Description

      This is an umbrella issue for FLIP-435: Introduce a New Materialized Table for Simplifying Data Pipelines, see FLIP design doc for more detail: https://cwiki.apache.org/confluence/display/FLINK/FLIP-435%3A+Introduce+a+New+Materialized+Table+for+Simplifying+Data+Pipelines

      Attachments

        Issue Links

          1.
          Introduce CatalogMaterializedTable and related interface to support materialized table Sub-task Resolved dalongliu
          2.
          Introduce test-filesystem Catalog based on FileSystem Connector to support materialized table Sub-task Resolved dalongliu
          3.
          Support create materialized table syntax Sub-task Resolved Feng Jin
          4.
          Support alter materialized table related syntaxes: suspend, resume, refresh, set and reset Sub-task Resolved Feng Jin
          5.
          Support drop materialized table syntax and execution in continuous refresh mode Sub-task Resolved Feng Jin
          6.
          Support describe job syntax and execution Sub-task Resolved xuyang
          7.
          Support the execution of create materialized table in continuous refresh mode Sub-task Resolved dalongliu
          8.
          Support the execution of suspend, resume materialized table in continuous refresh mode Sub-task Resolved Feng Jin
          9.
          Support the execution of refresh materialized table Sub-task Resolved xuyang
          10.
          Support the execution of create materialized table in full refresh mode Sub-task Resolved Feng Jin
          11.
          Support the execution of suspend, resume materialized table in full refresh mode Sub-task Resolved Feng Jin
          12.
          Support the execution of drop materialized table in full refresh mode Sub-task Resolved Feng Jin
          13.
          Support the execution of alter materialized table set freshness Sub-task Open Unassigned
          14.
          Support the execution of alter materialized table set refresh mode Sub-task Open Unassigned
          15.
          Support the execution of alter materialized table set options Sub-task Open Unassigned
          16.
          Support the execution of alter materialized table reset options Sub-task Open Unassigned
          17.
          Desc table syntax support materialized table Sub-task Open Unassigned
          18.
          Add doc for syntax `describe job 'xxx'` Sub-task Resolved xuyang
          19.
          Add end-to-end test for materialized table Sub-task Resolved Feng Jin
          20.
          Add materialized table statement doc Sub-task Resolved Feng Jin
          21.
          Add materialized table user guide doc Sub-task Open Unassigned
          22.
          Add materialized table quickstart doc Sub-task Resolved Feng Jin
          23.
          Add gateway refresh rest api doc Sub-task Resolved dalongliu
          24.
          Fix TestFileSystem Connector error setting execution mode Sub-task Resolved dalongliu
          25.
          Add materialized table develop guide docs Sub-task Open Unassigned
          26.
          Fix the bugs found in releasing testing of Materialized Table Sub-task Resolved dalongliu
          27.
          Remove hugo_extended package introduced by wrong operation Sub-task Resolved dalongliu
          28.
          Do not override the user-defined checkpoint interval in continuous mode. Sub-task Resolved Feng Jin

          Activity

            People

              lsy dalongliu
              lsy dalongliu
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated: