Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-2093

Partition Pruning doesn't handle Constant folding occasionally for BETWEEN clause.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.12.0, 0.11.1
    • Fix Version/s: 0.11.2
    • Component/s: Planner/Optimizer
    • Labels:
      None

      Description

      In some cases, Partition Pruning doesn't handle Constant folding for {{BETWEEN|| clause as shown below.

      default> CREATE EXTERNAL TABLE student (
        id        INT,
        name      TEXT,
        grade     TEXT,
        phone     TEXT
      ) USING TEXT
      PARTITION BY COLUMN (entrance_date TEXT)
      LOCATION '/data/student';
      
      
      default> \dfs -ls /data/student;
      Found 6 items
      drwx------   - tajo supergroup          0 2016-03-14 11:13 /data/student/entrance_date=20160308
      drwx------   - tajo supergroup          0 2016-03-14 11:13 /data/student/entrance_date=20160309
      drwx------   - tajo supergroup          0 2016-03-14 11:13 /data/student/entrance_date=20160310
      drwx------   - tajo supergroup          0 2016-03-14 11:13 /data/student/entrance_date=20160311
      drwx------   - tajo supergroup          0 2016-03-14 11:13 /data/student/entrance_date=20160312
      drwx------   - tajo supergroup          0 2016-03-14 11:13 /data/student/entrance_date=20160313
      
      default> \! date
      2016. 03. 14. (MON) 15:19:27 KST
      
      default> SELECT * FROM student WHERE entrance_date BETWEEN TO_CHAR(ADD_DAYS(CURRENT_DATE(), -2) , 'YYYYMMDD') AND TO_CHAR( ADD_DAYS(CURRENT_DATE(), -1) , 'YYYYMMDD');
      

      Above select query returns two rows as expected. But when I found logical plan on Tajo web UI, tajo retrieves all sub-directories of the example table as following:

      PARTITIONS_SCAN(4) on default.student
        => filter: default.student.entrance_date (TEXT) BETWEEN ASYMMETRIC to_char(add_days(current_date(),-2),YYYYMMDD) AND to_char(add_days(current_date(),-1),YYYYMMDD)
        => target list: default.student.id (INT4), default.student.name (TEXT), default.student.grade (TEXT), default.student.phone (TEXT), default.student.entrance_date (TEXT)
        => num of filtered paths: 6
        => out schema: {(5) default.student.id (INT4), default.student.name (TEXT), default.student.grade (TEXT), default.student.phone (TEXT), default.student.entrance_date (TEXT)}
        => in schema: {(4) default.student.id (INT4), default.student.name (TEXT), default.student.grade (TEXT), default.student.phone (TEXT)}
        => 0: hdfs://localhost:9010/data/student/entrance_date=20160308
        => 1: hdfs://localhost:9010/data/student/entrance_date=20160309
        => 2: hdfs://localhost:9010/data/student/entrance_date=20160310
        => 3: hdfs://localhost:9010/data/student/entrance_date=20160311
        => 4: hdfs://localhost:9010/data/student/entrance_date=20160312
        => 5: hdfs://localhost:9010/data/student/entrance_date=20160313
      

        Activity

        Hide
        githubbot ASF GitHub Bot added a comment -

        GitHub user blrunner opened a pull request:

        https://github.com/apache/tajo/pull/975

        TAJO-2093: Partition Pruning doesn't handle Constant folding occasionally for BETWEEN clause.

        You can merge this pull request into a Git repository by running:

        $ git pull https://github.com/blrunner/tajo TAJO-2093

        Alternatively you can review and apply these changes as the patch at:

        https://github.com/apache/tajo/pull/975.patch

        To close this pull request, make a commit to your master/trunk branch
        with (at least) the following in the commit message:

        This closes #975


        commit 26517dfc744232a39930429fd634ff023e821302
        Author: JaeHwa Jung <blrunner@apache.org>
        Date: 2016-03-14T06:32:35Z

        TAJO-2093: Partition Pruning doesn't handle Constant folding occasionally for BETWEEN clause.


        Show
        githubbot ASF GitHub Bot added a comment - GitHub user blrunner opened a pull request: https://github.com/apache/tajo/pull/975 TAJO-2093 : Partition Pruning doesn't handle Constant folding occasionally for BETWEEN clause. You can merge this pull request into a Git repository by running: $ git pull https://github.com/blrunner/tajo TAJO-2093 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/tajo/pull/975.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #975 commit 26517dfc744232a39930429fd634ff023e821302 Author: JaeHwa Jung <blrunner@apache.org> Date: 2016-03-14T06:32:35Z TAJO-2093 : Partition Pruning doesn't handle Constant folding occasionally for BETWEEN clause.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user hyunsik commented on the pull request:

        https://github.com/apache/tajo/pull/975#issuecomment-199568399

        LGTM +1

        Show
        githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on the pull request: https://github.com/apache/tajo/pull/975#issuecomment-199568399 LGTM +1
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Tajo-0.11.2-build #185 (See https://builds.apache.org/job/Tajo-0.11.2-build/185/)
        TAJO-2093: Partition Pruning doesn't handle Constant folding (blrunner: rev 3f994bf2efa0bd9b5b2c5fddfbc218ddec25a8c2)

        • tajo-core-tests/src/test/java/org/apache/tajo/engine/planner/TestPartitionedTableRewriter.java
        • tajo-plan/src/main/java/org/apache/tajo/plan/expr/AlgebraicUtil.java
        • tajo-plan/src/main/java/org/apache/tajo/plan/exprrewrite/rules/ConstantFolding.java
        • CHANGES
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Tajo-0.11.2-build #185 (See https://builds.apache.org/job/Tajo-0.11.2-build/185/ ) TAJO-2093 : Partition Pruning doesn't handle Constant folding (blrunner: rev 3f994bf2efa0bd9b5b2c5fddfbc218ddec25a8c2) tajo-core-tests/src/test/java/org/apache/tajo/engine/planner/TestPartitionedTableRewriter.java tajo-plan/src/main/java/org/apache/tajo/plan/expr/AlgebraicUtil.java tajo-plan/src/main/java/org/apache/tajo/plan/exprrewrite/rules/ConstantFolding.java CHANGES
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user blrunner closed the pull request at:

        https://github.com/apache/tajo/pull/975

        Show
        githubbot ASF GitHub Bot added a comment - Github user blrunner closed the pull request at: https://github.com/apache/tajo/pull/975

          People

          • Assignee:
            blrunner Jaehwa Jung
            Reporter:
            blrunner Jaehwa Jung
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development