Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-6326

add ProjectMergeRule at logical optimization stage

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.3.0
    • Component/s: Table API & SQL
    • Labels:
      None

      Description

      add ProjectMergeRule to merge projections. Some SQLs can not push projection into scan without this rule.
      e.g.

      table1: id: int, name: string
      table2: id: string, score: double, first: string, last: string

      SELECT a.id, b.score FROM 
       (SELECT id FROM table1 WHERE id > 10) a  
      LEFT OUTER JOIN 
       (SELECT * FROM table2) b 
      ON CAST(a.id AS VARCHAR) = b.id
      
      == Optimized Logical Plan without ProjectMergeRule ==
      DataSetCalc(select=[id, score])
        DataSetJoin(where=[=(id0, id1)], join=[id, id0, id1, score, first, last], joinType=[LeftOuterJoin])
          DataSetCalc(select=[id, CAST(id) AS id0], where=[>(id, 10)])
            BatchTableSourceScan(table=[[table1]], fields=[id])
          BatchTableSourceScan(table=[[table2]], fields=[id, score, first, last])
      
      == Optimized Logical Plan with ProjectMergeRule ==
      DataSetCalc(select=[id, score])
        DataSetJoin(where=[=(id0, id1)], join=[id, id0, id1, score], joinType=[LeftOuterJoin])
          DataSetCalc(select=[id, CAST(id) AS id0], where=[>(id, 10)])
            BatchTableSourceScan(table=[[table1]], fields=[id])
          BatchTableSourceScan(table=[[table2]], fields=[id, score])
      

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user godfreyhe opened a pull request:

          https://github.com/apache/flink/pull/3739

          FLINK-6326 [table] add ProjectMergeRule at logical optimization stage

          add ProjectMergeRule to merge projections

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/godfreyhe/flink FLINK-6326

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/flink/pull/3739.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #3739


          commit 1c14d02a4776becbc4147b576d5ee20dd8e15d24
          Author: godfreyhe <godfreyhe@163.com>
          Date: 2017-04-19T11:46:18Z

          FLINK-6326 [table] add ProjectMergeRule at logical optimization stage


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user godfreyhe opened a pull request: https://github.com/apache/flink/pull/3739 FLINK-6326 [table] add ProjectMergeRule at logical optimization stage add ProjectMergeRule to merge projections You can merge this pull request into a Git repository by running: $ git pull https://github.com/godfreyhe/flink FLINK-6326 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/3739.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3739 commit 1c14d02a4776becbc4147b576d5ee20dd8e15d24 Author: godfreyhe <godfreyhe@163.com> Date: 2017-04-19T11:46:18Z FLINK-6326 [table] add ProjectMergeRule at logical optimization stage
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user fhueske commented on a diff in the pull request:

          https://github.com/apache/flink/pull/3739#discussion_r112192763

          — Diff: flink-libraries/flink-table/src/test/scala/resources/testProjectMerge.out —
          @@ -0,0 +1,156 @@
          +== Abstract Syntax Tree ==
          — End diff –

          This file can be removed, IMO.

          Show
          githubbot ASF GitHub Bot added a comment - Github user fhueske commented on a diff in the pull request: https://github.com/apache/flink/pull/3739#discussion_r112192763 — Diff: flink-libraries/flink-table/src/test/scala/resources/testProjectMerge.out — @@ -0,0 +1,156 @@ +== Abstract Syntax Tree == — End diff – This file can be removed, IMO.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user fhueske commented on a diff in the pull request:

          https://github.com/apache/flink/pull/3739#discussion_r112192706

          — Diff: flink-libraries/flink-table/src/test/scala/org/apache/flink/table/api/scala/batch/ExplainTest.scala —
          @@ -119,4 +120,25 @@ class ExplainTest
          "../../src/test/scala/resources/testUnion1.out").mkString.replaceAll("\\r
          n", "\n")
          assertEquals(result, source)
          }
          +
          + @Test
          + def testProjectMerge(): Unit = {
          — End diff –

          I don't think we need to add a test to `ExplainTest.scala`. The other plan tests are sufficient to validate that the desired plans are generated.

          Show
          githubbot ASF GitHub Bot added a comment - Github user fhueske commented on a diff in the pull request: https://github.com/apache/flink/pull/3739#discussion_r112192706 — Diff: flink-libraries/flink-table/src/test/scala/org/apache/flink/table/api/scala/batch/ExplainTest.scala — @@ -119,4 +120,25 @@ class ExplainTest "../../src/test/scala/resources/testUnion1.out").mkString.replaceAll("\\r n", "\n") assertEquals(result, source) } + + @Test + def testProjectMerge(): Unit = { — End diff – I don't think we need to add a test to `ExplainTest.scala`. The other plan tests are sufficient to validate that the desired plans are generated.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user fhueske commented on the issue:

          https://github.com/apache/flink/pull/3739

          Merging

          Show
          githubbot ASF GitHub Bot added a comment - Github user fhueske commented on the issue: https://github.com/apache/flink/pull/3739 Merging
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/flink/pull/3739

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3739
          Hide
          fhueske Fabian Hueske added a comment -

          Fixed with 4024afffcd5a3f011e8ba2a3a1053e8bae6cec4f

          Show
          fhueske Fabian Hueske added a comment - Fixed with 4024afffcd5a3f011e8ba2a3a1053e8bae6cec4f

            People

            • Assignee:
              godfreyhe godfrey he
              Reporter:
              godfreyhe godfrey he
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development