Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-21091

Support ExecNode json serialization/deserialization

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Done
    • None
    • 1.14.0
    • Table SQL / Planner
    • None

    Description

      Since FLINK-20435 is finished, an ExecNode graph can be serialized to a json plan or a json plan can deserialized to an ExecNode graph. Only Blink planner introduce ExecNode, so legacy planner does not supports this feature.

      We will introduce the following methods to support json plan operations:

      // Get the json plan for the given statement.
String getJsonPlan(String stmt);

//  Returns the execution plan for the given json plan.
String explainJsonPlan(String jsonPlan, ExplainDetail... extraDetails);

//  Execute the given json plan, and return the execution result. 
TableResult executeJsonPlan(String jsonPlan);

      Currently, these methods are experimental features, so we introduce them into TableEnvironmentInternal as first step.

      This is an umbrella issue, we will create more related sub-tasks, including:
      1. introduce the above methods
      2. support json serialization/deserialization for all StreamExecNodes
      3. introduce complete testing to verify ExecNode serialization/deserialization for different queries

      Attachments

        1.
        		 Introduce getJsonPlan, explainJsonPlan and executeJsonPlan in TableEnvironmentInternal Sub-task Closed godfrey he
        2.
        		 Support StreamExecTableSource json serialization/deserialization Sub-task Closed godfrey he
        3.
        		 Support StreamExecSink json serialization/deserialization Sub-task Closed godfrey he
        4.
        		 Introduce ExecNodeGraphJsonPlanGenerator to serialize ExecNodeGraph to json plan and deserialize json plan to ExecNodeGraph Sub-task Closed godfrey he
        5.
        		 Introduce e2e test framework for json plan test Sub-task Closed Unassigned
        6.
        		 Support StreamExecCalc json serialization/deserialization Sub-task Closed godfrey he
        7.
        		 Support json serialization/deserialization for the push-down result of DynamicTableSource and DynamicTableSink Sub-task Closed godfrey he
        8.
        		 Support StreamExecGroupAggregate json serialization/deserialization Sub-task Closed godfrey he
        9.
        		 Introduce StateDeclaration Sub-task Closed Unassigned
        10.
        		 Support StreamExecWindowAggregate json serialization/deserialization Sub-task Closed godfrey he
        11.
        		 Support StreamExecMiniBatchAssigner json serialization/deserialization Sub-task Closed godfrey he
        12.
        		 Support StreamExecChangelogNormalize and StreamExecDropUpdateBefore json serde Sub-task Closed Wenlong Lyu
        13.
        		 Introduce ExecNodeTranslator Sub-task Closed godfrey he
        14.
        		 Support StreamExecDeduplicate json serialization/deserialization Sub-task Closed Terry Wang
        15.
        		 Support StreamExecWatermarkAssigner json serialization/deserialization Sub-task Closed godfrey he
        16.
        		 Support StreamExecCorrelate json serde Sub-task Closed Wenlong Lyu
        17.
        		 Support StreamExecLocalGroupAggregate and StreamExecGlobalGroupAggregate json serialization/deserialization Sub-task Closed godfrey he
        18.
        		 Support StreamExecRank and StreamExecSortLimit and StreamExecLimit Json ser/de Sub-task Closed Wenlong Lyu
        19.
        		 Support StreamExecJoin json serialization/deserialization Sub-task Closed Terry Wang
        20.
        		 Support StreamExecOverAggregate json ser/de Sub-task Closed Wenlong Lyu
        21.
        		 Support StreamExecUnion json ser/de Sub-task Closed Wenlong Lyu
        22.
        		 Support StreamExecMatch json ser/de Sub-task Closed Wenlong Lyu
        23.
        		 Support StreamExecIntervalJoin json ser/de Sub-task Closed Terry Wang
        24.
        		 Support StreamExecGroupWindowAggregate json ser/de Sub-task Closed Wenlong Lyu
        25.
        		 LogicalTypeJsonDeserializer/Serializer custom RowType/MapType/ArrayType/MultisetType Sub-task Closed Terry Wang
        26.
        		 Support StreamExecExpand json serialization/deserialization Sub-task Closed godfrey he
        27.
        		 Support StreamExecIncrementalGroupAggregate json serialization/deserialization Sub-task Closed godfrey he
        28.
        		 Support StreamExecTemporalJoin json serialization/deserialization Sub-task Closed Terry Wang
        29.
        		 Support StreamExecLookupJoin json serialization/deserialization Sub-task Closed Terry Wang
        30.
        		 Support StreamExecTemporalSort json serialization/deserialization Sub-task Closed godfrey he
        31.
        		 RelDataType of TableFunction different in local and CI Sub-task Closed Unassigned
        32.
        		 Check whether TemporalTableSourceSpec can be serialized or not Sub-task Closed godfrey he
        33.
        		 Reflections ClassNotFound when trying to deserialize a json plan in deployment envrionment Sub-task Closed Wenlong Lyu
        34.
        		 Reduce the ExecNode scan scope to improve performance when converting json plan to ExecNodeGraph Sub-task Closed godfrey he
        35.
        		 The ExecNode's id should always start from 1 in json plan test Sub-task Closed godfrey he
        36.
        		 The result of LogicalWindowJsonDeserializer is incorrect Sub-task Closed godfrey he
        37.
        		 Support StreamExecPythonCalc json serialization/deserialization Sub-task Resolved Huang Xingbo
        38.
        		 Support StreamExecPythonCorrelate json serialization/deserialization Sub-task Resolved Huang Xingbo
        39.
        		 Support StreamExecPythonGroupAggregate json serialization/deserialization Sub-task Resolved Huang Xingbo
        40.
        		 Support StreamExecPythonGroupWindowAggregate json serialization/deserialization Sub-task Resolved Huang Xingbo
        41.
        		 Support StreamExecPythonOverAggregate json serialization/deserialization Sub-task Resolved Huang Xingbo
        42.
        		 Support StreamExecValues json serialization/deserialization Sub-task Closed dalongliu

        Activity

          People

            godfreyhe godfrey he
            godfreyhe godfrey he
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: