Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-895

ConstEval should not be included in target list of projectable nodes

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.9.0
    • Component/s: Planner/Optimizer
    • Labels:
      None

      Description

      In some applications, aliased constant values can be used in WHERE, GROUP BY, HAVING, ORDER BY clauses. For those cases, current planner evaluates constant value in target list of projectable nodes.

      SELECT '1994' end as year, ... FROM lineitem WHERE group by year;
      

      This approach works well so far, but there are rooms for significant improvement.

      The main problem is that constant target requires many workaround code in NamedExprManager, and TargetListManager. As a result, it makes code complexity higher. The second problem is that many constant values evaluated in each row consume unnecessary I/O and network bandwidth for storing and transmitting.

      The solution seems to be simple. In logical planning phase, we should rewrite column references which actually indicates constant values.

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Tajo-master-build #334 (See https://builds.apache.org/job/Tajo-master-build/334/)
          TAJO-895: ConstEval should not be included in target list of projectable nodes. (hyunsik: rev 7e31a320144d2da6aae795f29d3bd9bbb16f9338)

          • tajo-core/src/test/resources/queries/TestJoinQuery/testLeftOuterJoinWithConstantExpr5.sql
          • tajo-core/src/test/resources/queries/TestJoinQuery/testLeftOuterJoinWithConstantExpr4.sql
          • tajo-core/src/test/resources/queries/TestGroupByQuery/testGroupByWithConstantKeys5.sql
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/LogicalPlanner.java
          • tajo-core/src/test/resources/results/TestGroupByQuery/testGroupByWithConstantKeys4.result
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/LogicalPlanPreprocessor.java
          • tajo-core/src/test/resources/results/TestGroupByQuery/testGroupByWithConstantKeys3.result
          • tajo-core/src/test/java/org/apache/tajo/engine/query/TestJoinQuery.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/PlannerUtil.java
          • tajo-core/src/test/resources/results/TestJoinQuery/testLeftOuterJoinWithConstantExpr5.result
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/TypeDeterminant.java
          • tajo-core/src/test/java/org/apache/tajo/engine/query/TestGroupByQuery.java
          • tajo-core/src/test/java/org/apache/tajo/engine/eval/TestEvalTreeUtil.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/rewrite/ProjectionPushDownRule.java
          • tajo-core/src/test/resources/queries/TestGroupByQuery/testGroupByWithConstantKeys3.sql
          • tajo-core/src/main/java/org/apache/tajo/engine/eval/FunctionEval.java
          • tajo-core/src/main/java/org/apache/tajo/engine/optimizer/eval/Prioritized.java
          • tajo-core/src/test/java/org/apache/tajo/engine/query/TestSortQuery.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/ExprNormalizer.java
          • tajo-core/src/test/resources/results/TestTajoCli/testLocalQueryWithoutFrom.result
          • tajo-core/src/test/resources/queries/TestWindowQuery/testWindow8.sql
          • tajo-core/src/main/java/org/apache/tajo/engine/optimizer/eval/EvalTreeOptimizationRule.java
          • tajo-core/src/test/resources/results/TestGroupByQuery/testGroupByWithConstantKeys5.result
          • tajo-core/src/main/java/org/apache/tajo/engine/eval/BinaryEval.java
          • tajo-core/src/test/resources/results/TestSortQuery/testSortWithConstKeys.result
          • tajo-core/src/test/resources/queries/TestGroupByQuery/testGroupByWithConstantKeys4.sql
          • tajo-core/src/test/resources/results/TestSelectQuery/testSelectAsterisk5.result
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/NamedExprsManager.java
          • tajo-core/src/main/java/org/apache/tajo/engine/optimizer/eval/rules/ConstantFolding.java
          • tajo-core/src/test/java/org/apache/tajo/engine/query/TestWindowQuery.java
          • tajo-core/src/test/resources/results/TestWindowQuery/testWindow8.result
          • tajo-core/src/main/java/org/apache/tajo/engine/optimizer/eval/rules/ConstantPropagation.java
          • tajo-core/src/test/resources/queries/TestSortQuery/testSortWithConstKeys.sql
          • tajo-core/src/main/java/org/apache/tajo/engine/optimizer/eval/EvalTreeOptimizer.java
          • CHANGES
          • tajo-core/src/test/java/org/apache/tajo/engine/query/TestSelectQuery.java
          • tajo-core/src/test/resources/results/TestSelectQuery/testSelectSameConstantsWithDifferentAliases3.result
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/ExprAnnotator.java
          • tajo-core/src/test/resources/results/TestJoinQuery/testLeftOuterJoinWithConstantExpr4.result
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/LogicalPlan.java
          • tajo-core/src/test/resources/queries/TestSelectQuery/testSelectSameConstantsWithDifferentAliases3.sql
          • tajo-core/src/test/java/org/apache/tajo/engine/planner/TestLogicalPlan.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Tajo-master-build #334 (See https://builds.apache.org/job/Tajo-master-build/334/ ) TAJO-895 : ConstEval should not be included in target list of projectable nodes. (hyunsik: rev 7e31a320144d2da6aae795f29d3bd9bbb16f9338) tajo-core/src/test/resources/queries/TestJoinQuery/testLeftOuterJoinWithConstantExpr5.sql tajo-core/src/test/resources/queries/TestJoinQuery/testLeftOuterJoinWithConstantExpr4.sql tajo-core/src/test/resources/queries/TestGroupByQuery/testGroupByWithConstantKeys5.sql tajo-core/src/main/java/org/apache/tajo/engine/planner/LogicalPlanner.java tajo-core/src/test/resources/results/TestGroupByQuery/testGroupByWithConstantKeys4.result tajo-core/src/main/java/org/apache/tajo/engine/planner/LogicalPlanPreprocessor.java tajo-core/src/test/resources/results/TestGroupByQuery/testGroupByWithConstantKeys3.result tajo-core/src/test/java/org/apache/tajo/engine/query/TestJoinQuery.java tajo-core/src/main/java/org/apache/tajo/engine/planner/PlannerUtil.java tajo-core/src/test/resources/results/TestJoinQuery/testLeftOuterJoinWithConstantExpr5.result tajo-core/src/main/java/org/apache/tajo/engine/planner/TypeDeterminant.java tajo-core/src/test/java/org/apache/tajo/engine/query/TestGroupByQuery.java tajo-core/src/test/java/org/apache/tajo/engine/eval/TestEvalTreeUtil.java tajo-core/src/main/java/org/apache/tajo/engine/planner/rewrite/ProjectionPushDownRule.java tajo-core/src/test/resources/queries/TestGroupByQuery/testGroupByWithConstantKeys3.sql tajo-core/src/main/java/org/apache/tajo/engine/eval/FunctionEval.java tajo-core/src/main/java/org/apache/tajo/engine/optimizer/eval/Prioritized.java tajo-core/src/test/java/org/apache/tajo/engine/query/TestSortQuery.java tajo-core/src/main/java/org/apache/tajo/engine/planner/ExprNormalizer.java tajo-core/src/test/resources/results/TestTajoCli/testLocalQueryWithoutFrom.result tajo-core/src/test/resources/queries/TestWindowQuery/testWindow8.sql tajo-core/src/main/java/org/apache/tajo/engine/optimizer/eval/EvalTreeOptimizationRule.java tajo-core/src/test/resources/results/TestGroupByQuery/testGroupByWithConstantKeys5.result tajo-core/src/main/java/org/apache/tajo/engine/eval/BinaryEval.java tajo-core/src/test/resources/results/TestSortQuery/testSortWithConstKeys.result tajo-core/src/test/resources/queries/TestGroupByQuery/testGroupByWithConstantKeys4.sql tajo-core/src/test/resources/results/TestSelectQuery/testSelectAsterisk5.result tajo-core/src/main/java/org/apache/tajo/engine/planner/NamedExprsManager.java tajo-core/src/main/java/org/apache/tajo/engine/optimizer/eval/rules/ConstantFolding.java tajo-core/src/test/java/org/apache/tajo/engine/query/TestWindowQuery.java tajo-core/src/test/resources/results/TestWindowQuery/testWindow8.result tajo-core/src/main/java/org/apache/tajo/engine/optimizer/eval/rules/ConstantPropagation.java tajo-core/src/test/resources/queries/TestSortQuery/testSortWithConstKeys.sql tajo-core/src/main/java/org/apache/tajo/engine/optimizer/eval/EvalTreeOptimizer.java CHANGES tajo-core/src/test/java/org/apache/tajo/engine/query/TestSelectQuery.java tajo-core/src/test/resources/results/TestSelectQuery/testSelectSameConstantsWithDifferentAliases3.result tajo-core/src/main/java/org/apache/tajo/engine/planner/ExprAnnotator.java tajo-core/src/test/resources/results/TestJoinQuery/testLeftOuterJoinWithConstantExpr4.result tajo-core/src/main/java/org/apache/tajo/engine/planner/LogicalPlan.java tajo-core/src/test/resources/queries/TestSelectQuery/testSelectSameConstantsWithDifferentAliases3.sql tajo-core/src/test/java/org/apache/tajo/engine/planner/TestLogicalPlan.java
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/tajo/pull/111

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/tajo/pull/111
          Hide
          hyunsik Hyunsik Choi added a comment -

          committed to master branch. Fixing it as resolved.

          Show
          hyunsik Hyunsik Choi added a comment - committed to master branch. Fixing it as resolved.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jinossy commented on the pull request:

          https://github.com/apache/tajo/pull/111#issuecomment-52008894

          +1
          Looks great to me. I've verified in real cluster

          Show
          githubbot ASF GitHub Bot added a comment - Github user jinossy commented on the pull request: https://github.com/apache/tajo/pull/111#issuecomment-52008894 +1 Looks great to me. I've verified in real cluster
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jinossy commented on the pull request:

          https://github.com/apache/tajo/pull/111#issuecomment-52006420

          Sorry too late I'm reviewing

          Show
          githubbot ASF GitHub Bot added a comment - Github user jinossy commented on the pull request: https://github.com/apache/tajo/pull/111#issuecomment-52006420 Sorry too late I'm reviewing
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on the pull request:

          https://github.com/apache/tajo/pull/111#issuecomment-52006102

          This patch is conflict to TAJO-906. For next work, I need this work before TAJO-906. It would greatly appreciate if someone review this.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on the pull request: https://github.com/apache/tajo/pull/111#issuecomment-52006102 This patch is conflict to TAJO-906 . For next work, I need this work before TAJO-906 . It would greatly appreciate if someone review this.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on the pull request:

          https://github.com/apache/tajo/pull/111#issuecomment-51760307

          I've fixed the bug I mentioned.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on the pull request: https://github.com/apache/tajo/pull/111#issuecomment-51760307 I've fixed the bug I mentioned.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on the pull request:

          https://github.com/apache/tajo/pull/111#issuecomment-51757343

          This has some bug for window function.

          The reproduction query is as follows:
          ```
          select l_orderkey, l_quantity, rank() over (partition by l_orderkey) as r, 5 as const_val from lineitem;
          ```

          This problem is caused by handling const values in a window clause. I'll submit the patch soon, including the fix of the problem.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on the pull request: https://github.com/apache/tajo/pull/111#issuecomment-51757343 This has some bug for window function. The reproduction query is as follows: ``` select l_orderkey, l_quantity, rank() over (partition by l_orderkey) as r, 5 as const_val from lineitem; ``` This problem is caused by handling const values in a window clause. I'll submit the patch soon, including the fix of the problem.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on the pull request:

          https://github.com/apache/tajo/pull/111#issuecomment-51754991

          I've rebased it against the latest revision.

          Could you anyone review this?

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on the pull request: https://github.com/apache/tajo/pull/111#issuecomment-51754991 I've rebased it against the latest revision. Could you anyone review this?
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user hyunsik opened a pull request:

          https://github.com/apache/tajo/pull/111

          TAJO-895: ConstEval should not be included in target list of projectable...

          ... nodes.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/hyunsik/tajo TAJO-895

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/tajo/pull/111.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #111


          commit f9f90e570abfb31e0139c96d15da0b8f0f042339
          Author: Hyunsik Choi <hyunsik@apache.org>
          Date: 2014-08-08T09:13:49Z

          TAJO-895: ConstEval should not be included in target list of projectable nodes.


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user hyunsik opened a pull request: https://github.com/apache/tajo/pull/111 TAJO-895 : ConstEval should not be included in target list of projectable... ... nodes. You can merge this pull request into a Git repository by running: $ git pull https://github.com/hyunsik/tajo TAJO-895 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/tajo/pull/111.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #111 commit f9f90e570abfb31e0139c96d15da0b8f0f042339 Author: Hyunsik Choi <hyunsik@apache.org> Date: 2014-08-08T09:13:49Z TAJO-895 : ConstEval should not be included in target list of projectable nodes.

            People

            • Assignee:
              hyunsik Hyunsik Choi
              Reporter:
              hyunsik Hyunsik Choi
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development