Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-539

Change some EvalNode::eval to directly return a Datum value

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.0
    • Component/s: Expression
    • Labels:
      None

      Description

      After TAJO-501, we can ensure that all expressions except for AggregationFunctionCallEval can be evaluated by calling once eval() instead of calling eval followed by terminate. In addition, current EvalNode implementation involves unnecessary memory consumption to keep EvalContext for all expressions which are not even aggregation eval.

      If we change EvalNode::eval() to directly return a Datum value, it would reduce the memory consumption and CPU costs.

      1. TAJO-539_20140125_15:13:44.patch
        180 kB
        Hyunsik Choi
      2. TAJO-539.patch
        180 kB
        Hyunsik Choi

        Activity

        Hide
        hudson Hudson added a comment -

        ABORTED: Integrated in Tajo-master-build #29 (See https://builds.apache.org/job/Tajo-master-build/29/)
        TAJO-539: Change some EvalNode::eval to directly return a Datum value. (hyunsik: https://git-wip-us.apache.org/repos/asf?p=incubator-tajo.git&a=commit&h=e23e78ccd4c740fd93d2629e5b523c3f4b29c199)

        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/HavingExec.java
        • tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/eval/ExprTestBase.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/PreLogicalPlanVerifier.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/rewrite/ProjectionPushDownRule.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/MergeFullOuterJoinExec.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/HashLeftAntiJoinExec.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/eval/BinaryEval.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/rewrite/PartitionedTableRewriter.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/eval/BetweenPredicateEval.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/AggregationExec.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/eval/CastEval.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/GroupbyNode.java
        • tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/planner/physical/TestPhysicalPlanner.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/BNLJoinExec.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/HashAggregateExec.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/eval/AggregationFunctionCallEval.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/eval/FunctionEval.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/eval/AlgebraicUtil.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/eval/NotEval.java
        • tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/eval/TestEvalTreeUtil.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/EvalExprExec.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/LogicalPlanner.java
        • tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/eval/TestEvalTree.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/eval/RowConstantEval.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/eval/CaseWhenEval.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/eval/ConstEval.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/HashLeftSemiJoinExec.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/GlobalEngine.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/eval/GeneralFunctionEval.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/SortAggregateExec.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/eval/EvalNode.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/eval/SignedEval.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/LogicalPlanVerifier.java
        • CHANGES.txt
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/eval/PartialBinaryExpr.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/eval/IsNullEval.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/HashFullOuterJoinExec.java
        • tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/planner/TestPlannerUtil.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/ExprsVerifier.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/MergeJoinExec.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/global/GlobalPlanner.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/utils/TupleUtil.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/eval/InEval.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/SeqScanExec.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/RightOuterMergeJoinExec.java
        • tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/planner/TestLogicalPlanner.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/PlannerUtil.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/NLLeftOuterJoinExec.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/HashLeftOuterJoinExec.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/HashJoinExec.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/eval/EvalContext.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/Projector.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/BSTIndexScanExec.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/eval/PatternMatchPredicateEval.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/eval/FieldEval.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/NLJoinExec.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/eval/EvalTreeUtil.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/ProjectionExec.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/SelectionExec.java
        Show
        hudson Hudson added a comment - ABORTED: Integrated in Tajo-master-build #29 (See https://builds.apache.org/job/Tajo-master-build/29/ ) TAJO-539 : Change some EvalNode::eval to directly return a Datum value. (hyunsik: https://git-wip-us.apache.org/repos/asf?p=incubator-tajo.git&a=commit&h=e23e78ccd4c740fd93d2629e5b523c3f4b29c199 ) tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/HavingExec.java tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/eval/ExprTestBase.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/PreLogicalPlanVerifier.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/rewrite/ProjectionPushDownRule.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/MergeFullOuterJoinExec.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/HashLeftAntiJoinExec.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/eval/BinaryEval.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/rewrite/PartitionedTableRewriter.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/eval/BetweenPredicateEval.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/AggregationExec.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/eval/CastEval.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/GroupbyNode.java tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/planner/physical/TestPhysicalPlanner.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/BNLJoinExec.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/HashAggregateExec.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/eval/AggregationFunctionCallEval.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/eval/FunctionEval.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/eval/AlgebraicUtil.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/eval/NotEval.java tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/eval/TestEvalTreeUtil.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/EvalExprExec.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/LogicalPlanner.java tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/eval/TestEvalTree.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/eval/RowConstantEval.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/eval/CaseWhenEval.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/eval/ConstEval.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/HashLeftSemiJoinExec.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/GlobalEngine.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/eval/GeneralFunctionEval.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/SortAggregateExec.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/eval/EvalNode.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/eval/SignedEval.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/LogicalPlanVerifier.java CHANGES.txt tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/eval/PartialBinaryExpr.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/eval/IsNullEval.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/HashFullOuterJoinExec.java tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/planner/TestPlannerUtil.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/ExprsVerifier.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/MergeJoinExec.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/global/GlobalPlanner.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/utils/TupleUtil.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/eval/InEval.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/SeqScanExec.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/RightOuterMergeJoinExec.java tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/planner/TestLogicalPlanner.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/PlannerUtil.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/NLLeftOuterJoinExec.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/HashLeftOuterJoinExec.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/HashJoinExec.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/eval/EvalContext.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/Projector.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/BSTIndexScanExec.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/eval/PatternMatchPredicateEval.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/eval/FieldEval.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/NLJoinExec.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/eval/EvalTreeUtil.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/ProjectionExec.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/SelectionExec.java
        Hide
        hyunsik Hyunsik Choi added a comment -

        It got +1 in RB. committed it to master branch.

        Show
        hyunsik Hyunsik Choi added a comment - It got +1 in RB. committed it to master branch.
        Hide
        hyunsik Hyunsik Choi added a comment -

        Updated the review request against branch master in reviewboard
        https://reviews.apache.org/r/17308/

        Show
        hyunsik Hyunsik Choi added a comment - Updated the review request against branch master in reviewboard https://reviews.apache.org/r/17308/
        Hide
        hyunsik Hyunsik Choi added a comment -

        According to my benchmark, it significantly reduces memory consumption, and GC time. For example, it reduces the response time of TPC-H Q3 by 15%, and it also reduces by 50% the response time of heavy aggregation query that processes 600 million distinct grouping keys.

        Show
        hyunsik Hyunsik Choi added a comment - According to my benchmark, it significantly reduces memory consumption, and GC time. For example, it reduces the response time of TPC-H Q3 by 15%, and it also reduces by 50% the response time of heavy aggregation query that processes 600 million distinct grouping keys.
        Hide
        hyunsik Hyunsik Choi added a comment -

        Created a review request against branch master in reviewboard
        https://reviews.apache.org/r/17308/

        Show
        hyunsik Hyunsik Choi added a comment - Created a review request against branch master in reviewboard https://reviews.apache.org/r/17308/

          People

          • Assignee:
            hyunsik Hyunsik Choi
            Reporter:
            hyunsik Hyunsik Choi
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development