Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-415

Some complex queries causes NPE and unlimited recursions.

    Details

      Description

      This patch fixes the following critical bugs.

      • GreedyHeuristicJoinOrder::getCost may cause unlimited recursion.
        • getCost() method is a recursion function.
        • getCost() method has not considered all logical operators, so some logical operators are processed in a default case which is implemented incorrectly. It causes unlimited recursion.
      • ProjectionPushDown rewrite rule tries to push down necessary target lists to all subqueries.
        • It may cause some problem in a subquery which includes a group-by operator because there is no unnecessary target lists in the subquery, including a group-by operator. This patch prevents ProjectionPushDown to push down target lists to some subqueries.
      • Task::setLogicalPlan does not traverse descendant nodes of TableSubQueryNode.
        • So, it have missed some logical operators.
      • SubQuery::getInputVolume may cause NPE when a logical plan includes a bushy join tree.
        • In the past, a distributed plan is generated from left-deep join tree. At that time, the volume of join relations are computed directly from joined results (left) or relation nodes (right). However, for bushy join, getInputVolume() has some cases to estimate intermediate data volume prior to actual joins.
      • FilterPushDown tries to validate outer join's null suppression. This code assumes that outer join's input relations are only ScanNodes.
        • Actually, outer join's input relations can be either ScanNode or TableSubQueryNode.
      1. TAJO-415.patch
        17 kB
        Hyunsik Choi

        Activity

        Hide
        jihoonson Jihoon Son added a comment -

        Thanks.
        It will be very interesting!

        Show
        jihoonson Jihoon Son added a comment - Thanks. It will be very interesting!
        Hide
        hyunsik Hyunsik Choi added a comment -

        I totally agree with the needs for more elaborate ways to estimate intermediate data volume. Later, I'll create a jira issue for that.

        Show
        hyunsik Hyunsik Choi added a comment - I totally agree with the needs for more elaborate ways to estimate intermediate data volume. Later, I'll create a jira issue for that.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Tajo-trunk-postcommit #619 (See https://builds.apache.org/job/Tajo-trunk-postcommit/619/)
        TAJO-415: Some complex queries causes NPE and unlimited recursions. (hyunsik) (hyunsik: https://git-wip-us.apache.org/repos/asf?p=incubator-tajo.git&a=commit&h=a90895a74a13c654a8d12345bbbdd8d3b08c604c)

        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/eval/FieldEval.java
        • tajo-core/tajo-core-backend/src/test/queries/tajo415_case.sql
        • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/FunctionDesc.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/SubQuery.java
        • CHANGES.txt
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/rewrite/ProjectionPushDownRule.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/rewrite/FilterPushDownRule.java
        • tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/query/TestCaseByCases.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryUnit.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/join/GreedyHeuristicJoinOrderAlgorithm.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Tajo-trunk-postcommit #619 (See https://builds.apache.org/job/Tajo-trunk-postcommit/619/ ) TAJO-415 : Some complex queries causes NPE and unlimited recursions. (hyunsik) (hyunsik: https://git-wip-us.apache.org/repos/asf?p=incubator-tajo.git&a=commit&h=a90895a74a13c654a8d12345bbbdd8d3b08c604c ) tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/eval/FieldEval.java tajo-core/tajo-core-backend/src/test/queries/tajo415_case.sql tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/FunctionDesc.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/SubQuery.java CHANGES.txt tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/rewrite/ProjectionPushDownRule.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/rewrite/FilterPushDownRule.java tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/query/TestCaseByCases.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryUnit.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/join/GreedyHeuristicJoinOrderAlgorithm.java
        Hide
        hyunsik Hyunsik Choi added a comment -

        committed the bug fix to master. Thanks for the review.

        Show
        hyunsik Hyunsik Choi added a comment - committed the bug fix to master. Thanks for the review.
        Hide
        jihoonson Jihoon Son added a comment -

        +1 for the patch.
        Later, we need to improve the size estimation of the intermediate data.

        Show
        jihoonson Jihoon Son added a comment - +1 for the patch. Later, we need to improve the size estimation of the intermediate data.

          People

          • Assignee:
            hyunsik Hyunsik Choi
            Reporter:
            hyunsik Hyunsik Choi
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development