Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-283 Add Table Partitioning
  3. TAJO-338

Add Query Optimization Part for Column-Partitioned Tables

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 0.8.0
    • Component/s: Storage
    • Labels:
      None

      Description

      If user write partitioned column at where clause, Tajo must make a plan to scan a partition directory path. So, Planner needs to improve for partitioned table.

      1. TAJO-338_2.patch
        216 kB
        Hyunsik Choi
      2. TAJO-338.patch
        213 kB
        Hyunsik Choi

        Activity

        Hide
        hyunsik Hyunsik Choi added a comment -

        In order to test the processing of partitioned tables, I needed table insertions prior to test. Table insertions are duplicated to some unit tests in TestInsertQuery. So, I integrated them.

        Show
        hyunsik Hyunsik Choi added a comment - In order to test the processing of partitioned tables, I needed table insertions prior to test. Table insertions are duplicated to some unit tests in TestInsertQuery. So, I integrated them.
        Hide
        hyunsik Hyunsik Choi added a comment -

        I moved some unit tests related to partitioned table from TestInsertQuery to TestTablePartitions.

        Show
        hyunsik Hyunsik Choi added a comment - I moved some unit tests related to partitioned table from TestInsertQuery to TestTablePartitions.
        Hide
        coderplay Min Zhou added a comment -

        Why did you remove some testcases which insert overwrite a partitioned table in TestInsertQuery.java? Any conflict against this issue?

        see

        git show f58f6ee827139c01253902f0b1076b47fde9d5e6 ./tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/query/TestInsertQuery.java
        
        Show
        coderplay Min Zhou added a comment - Why did you remove some testcases which insert overwrite a partitioned table in TestInsertQuery.java? Any conflict against this issue? see git show f58f6ee827139c01253902f0b1076b47fde9d5e6 ./tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/query/TestInsertQuery.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Tajo-trunk-postcommit #628 (See https://builds.apache.org/job/Tajo-trunk-postcommit/628/)
        TAJO-338 - Add Query Optimization Part for Column-Partitioned Tables. (hyunsik) (hyunsik: https://git-wip-us.apache.org/repos/asf?p=incubator-tajo.git&a=commit&h=f58f6ee827139c01253902f0b1076b47fde9d5e6)

        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/DefaultTaskScheduler.java
        • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/DDLBuilder.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/CreateTableNode.java
        • tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/eval/ExprTestBase.java
        • tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/worker/TestRangeRetrieverHandler.java
        • tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/planner/physical/TestSortExec.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/TajoMasterClientService.java
        • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/partition/Partitions.java
        • tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/planner/physical/TestPhysicalPlanner.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/PhysicalPlannerImpl.java
        • tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/util/TestTupleUtil.java
        • tajo-common/src/main/java/org/apache/tajo/datum/BooleanDatum.java
        • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/Schema.java
        • tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/planner/TestLogicalPlan.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/GlobalEngine.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/parser/SQLAnalyzer.java
        • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/TableDesc.java
        • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java
        • tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogStore.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/eval/EvalTreeUtil.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/Target.java
        • tajo-core/tajo-core-backend/src/main/proto/ClientProtos.proto
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/BasicLogicalPlanVisitor.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/TableSubQueryNode.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/rewrite/ProjectionPushDownRule.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/StoreTableNode.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/eval/InEval.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryMasterTask.java
        • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/partition/PartitionDesc.java
        • tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/query/TestTablePartitions.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/LogicalPlanner.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/NodeType.java
        • tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/parser/TestSQLAnalyzer.java
        • tajo-common/src/main/java/org/apache/tajo/datum/DatumFactory.java
        • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/DerbyStore.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/utils/TupleUtil.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/RelationNode.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/ScanNode.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/eval/LikePredicateEval.java
        • tajo-core/tajo-core-backend/src/test/queries/create_table_partition_by_column.sql
        • tajo-core/tajo-core-backend/src/main/antlr4/org/apache/tajo/engine/parser/SQLLexer.g4
        • tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/planner/physical/TestHashAntiJoinExec.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/rewrite/FilterPushDownRule.java
        • tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/master/TestExecutionBlockCursor.java
        • tajo-catalog/tajo-catalog-client/src/main/java/org/apache/tajo/catalog/AbstractCatalogClient.java
        • tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/planner/physical/TestHashSemiJoinExec.java
        • tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/planner/TestLogicalOptimizer.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/global/ExecutionBlock.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/PhysicalExec.java
        • tajo-catalog/tajo-catalog-common/src/test/java/org/apache/tajo/catalog/TestSchema.java
        • tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/query/TestInsertQuery.java
        • tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/planner/physical/TestBSTIndexExec.java
        • tajo-common/src/main/java/org/apache/tajo/util/TUtil.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/global/GlobalPlanner.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/rewrite/PartitionedTableRewriter.java
        • tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalog.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/LogicalOptimizer.java
        • tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/eval/TestEvalTreeUtil.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/PartitionedTableScanNode.java
        • tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/client/TestTajoClient.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/ColumnPartitionedTableStoreExec.java
        • tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestDBStore.java
        • tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/master/TestGlobalPlanner.java
        • tajo-core/tajo-core-backend/src/main/antlr4/org/apache/tajo/engine/parser/SQLParser.g4
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/eval/AlgebraicUtil.java
        • tajo-algebra/src/main/java/org/apache/tajo/algebra/CreateTable.java
        • tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/join/GreedyHeuristicJoinOrderAlgorithm.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/SubQuery.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/SeqScanExec.java
        • CHANGES.txt
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/PlannerUtil.java
        • tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/test/java/org/apache/tajo/catalog/store/TestHCatalogStore.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/join/JoinGraph.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/LogicalPlanVisitor.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Tajo-trunk-postcommit #628 (See https://builds.apache.org/job/Tajo-trunk-postcommit/628/ ) TAJO-338 - Add Query Optimization Part for Column-Partitioned Tables. (hyunsik) (hyunsik: https://git-wip-us.apache.org/repos/asf?p=incubator-tajo.git&a=commit&h=f58f6ee827139c01253902f0b1076b47fde9d5e6 ) tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/DefaultTaskScheduler.java tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/DDLBuilder.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/CreateTableNode.java tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/eval/ExprTestBase.java tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/worker/TestRangeRetrieverHandler.java tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/planner/physical/TestSortExec.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/TajoMasterClientService.java tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/partition/Partitions.java tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/planner/physical/TestPhysicalPlanner.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/PhysicalPlannerImpl.java tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/util/TestTupleUtil.java tajo-common/src/main/java/org/apache/tajo/datum/BooleanDatum.java tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/Schema.java tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/planner/TestLogicalPlan.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/GlobalEngine.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/parser/SQLAnalyzer.java tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/TableDesc.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogStore.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/eval/EvalTreeUtil.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/Target.java tajo-core/tajo-core-backend/src/main/proto/ClientProtos.proto tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/BasicLogicalPlanVisitor.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/TableSubQueryNode.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/rewrite/ProjectionPushDownRule.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/StoreTableNode.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/eval/InEval.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryMasterTask.java tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/partition/PartitionDesc.java tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/query/TestTablePartitions.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/LogicalPlanner.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/NodeType.java tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/parser/TestSQLAnalyzer.java tajo-common/src/main/java/org/apache/tajo/datum/DatumFactory.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/DerbyStore.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/utils/TupleUtil.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/RelationNode.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/ScanNode.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/eval/LikePredicateEval.java tajo-core/tajo-core-backend/src/test/queries/create_table_partition_by_column.sql tajo-core/tajo-core-backend/src/main/antlr4/org/apache/tajo/engine/parser/SQLLexer.g4 tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/planner/physical/TestHashAntiJoinExec.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/rewrite/FilterPushDownRule.java tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/master/TestExecutionBlockCursor.java tajo-catalog/tajo-catalog-client/src/main/java/org/apache/tajo/catalog/AbstractCatalogClient.java tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/planner/physical/TestHashSemiJoinExec.java tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/planner/TestLogicalOptimizer.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/global/ExecutionBlock.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/PhysicalExec.java tajo-catalog/tajo-catalog-common/src/test/java/org/apache/tajo/catalog/TestSchema.java tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/query/TestInsertQuery.java tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/planner/physical/TestBSTIndexExec.java tajo-common/src/main/java/org/apache/tajo/util/TUtil.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/global/GlobalPlanner.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/rewrite/PartitionedTableRewriter.java tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalog.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/LogicalOptimizer.java tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/eval/TestEvalTreeUtil.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/PartitionedTableScanNode.java tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/client/TestTajoClient.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/ColumnPartitionedTableStoreExec.java tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestDBStore.java tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/master/TestGlobalPlanner.java tajo-core/tajo-core-backend/src/main/antlr4/org/apache/tajo/engine/parser/SQLParser.g4 tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/eval/AlgebraicUtil.java tajo-algebra/src/main/java/org/apache/tajo/algebra/CreateTable.java tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/join/GreedyHeuristicJoinOrderAlgorithm.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/SubQuery.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/SeqScanExec.java CHANGES.txt tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/PlannerUtil.java tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/test/java/org/apache/tajo/catalog/store/TestHCatalogStore.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/join/JoinGraph.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/LogicalPlanVisitor.java
        Hide
        hyunsik Hyunsik Choi added a comment -

        committed the latest patch to master. Thanks for the review.

        Show
        hyunsik Hyunsik Choi added a comment - committed the latest patch to master. Thanks for the review.
        Hide
        blrunner Jaehwa Jung added a comment -

        +1 for the patch.
        'mvn clean install -Phcatalog-0.12.0' finished successfully.
        Ship it.

        Show
        blrunner Jaehwa Jung added a comment - +1 for the patch. 'mvn clean install -Phcatalog-0.12.0' finished successfully. Ship it.
        Hide
        hyunsik Hyunsik Choi added a comment -

        I've updated the second patch.

        Show
        hyunsik Hyunsik Choi added a comment - I've updated the second patch.
        Hide
        blrunner Jaehwa Jung added a comment -

        Awesome!
        It looks great.
        But you have to update HCatalogStore because It converted Partitions to PartitionDesc.
        You can find build failure using HCatalogStore as follows:

        *mvn clean install -Phcatalog-0.12.0*
        [INFO] ------------------------------------------------------------------------
        [INFO] BUILD FAILURE
        [INFO] ------------------------------------------------------------------------
        [INFO] Total time: 3:49.379s
        [INFO] Finished at: Thu Dec 19 10:47:28 KST 2013
        [INFO] Final Memory: 43M/123M
        [INFO] ------------------------------------------------------------------------
        [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.5.1:compile (default-compile) on project tajo-hcatalog: Compilation failure: Compilation failure:
        [ERROR] /Users/blrunner/IdeaProjects/TAJO-338/tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogStore.java:[33,40] cannot find symbol
        [ERROR] symbol  : class Partitions
        [ERROR] location: package org.apache.tajo.catalog.partition
        [ERROR] /Users/blrunner/IdeaProjects/TAJO-338/tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogStore.java:[109,4] cannot find symbol
        [ERROR] symbol  : class Partitions
        [ERROR] location: class org.apache.tajo.catalog.store.HCatalogStore
        [ERROR] /Users/blrunner/IdeaProjects/TAJO-338/tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogStore.java:[181,27] cannot find symbol
        [ERROR] symbol  : class Partitions
        [ERROR] location: class org.apache.tajo.catalog.store.HCatalogStore
        
        
        Show
        blrunner Jaehwa Jung added a comment - Awesome! It looks great. But you have to update HCatalogStore because It converted Partitions to PartitionDesc. You can find build failure using HCatalogStore as follows: *mvn clean install -Phcatalog-0.12.0* [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 3:49.379s [INFO] Finished at: Thu Dec 19 10:47:28 KST 2013 [INFO] Final Memory: 43M/123M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.5.1:compile (default-compile) on project tajo-hcatalog: Compilation failure: Compilation failure: [ERROR] /Users/blrunner/IdeaProjects/TAJO-338/tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogStore.java:[33,40] cannot find symbol [ERROR] symbol : class Partitions [ERROR] location: package org.apache.tajo.catalog.partition [ERROR] /Users/blrunner/IdeaProjects/TAJO-338/tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogStore.java:[109,4] cannot find symbol [ERROR] symbol : class Partitions [ERROR] location: class org.apache.tajo.catalog.store.HCatalogStore [ERROR] /Users/blrunner/IdeaProjects/TAJO-338/tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogStore.java:[181,27] cannot find symbol [ERROR] symbol : class Partitions [ERROR] location: class org.apache.tajo.catalog.store.HCatalogStore
        Hide
        hyunsik Hyunsik Choi added a comment -

        This patch does as follows:

        • Add PartitionedTableScanNode
        • Add a rewrite rule named PartitionedTableRewriter
          • First of all, PartitionedTableRewriter finds partition predicates from search condition.
            • A partition predicate must be a simple predicate, meaning one column reference and one constant with one of =, <, <=, >, >=, IN, and LIKE predicate operators.
          • PartitionedTableRewriter also finds necessary partition paths via PathFilters generated from partition predicates.
          • Then, it rewrites ScanNode to PartitionedTableScanNode with filtered partition paths.
        • Some methods in EvalTreeUtil and AlgebraUtil are refactored and renamed to more comprehensive names.
        • Rename Partitions to PartitionDesc
        • Add some schema rewrite methods to ColumnPartitionedTableStoreExec and SeqScan
          • Column Partitioned Table aims at hive-compatible table partition.
          • As a result, scan does not read field values corresponding to partition key columns from files, and store does not store field values to files.
          • Instead of, field values corresponding to partition key columns are just retrieved from a partition path.
          • For that, the rewriter usually removes partition key columns from actual input schema of scan plan and output schema of store plan.
        Show
        hyunsik Hyunsik Choi added a comment - This patch does as follows: Add PartitionedTableScanNode Add a rewrite rule named PartitionedTableRewriter First of all, PartitionedTableRewriter finds partition predicates from search condition. A partition predicate must be a simple predicate, meaning one column reference and one constant with one of =, <, <=, >, >=, IN, and LIKE predicate operators. PartitionedTableRewriter also finds necessary partition paths via PathFilters generated from partition predicates. Then, it rewrites ScanNode to PartitionedTableScanNode with filtered partition paths. Some methods in EvalTreeUtil and AlgebraUtil are refactored and renamed to more comprehensive names. Rename Partitions to PartitionDesc Add some schema rewrite methods to ColumnPartitionedTableStoreExec and SeqScan Column Partitioned Table aims at hive-compatible table partition. As a result, scan does not read field values corresponding to partition key columns from files, and store does not store field values to files. Instead of, field values corresponding to partition key columns are just retrieved from a partition path. For that, the rewriter usually removes partition key columns from actual input schema of scan plan and output schema of store plan.
        Hide
        blrunner Jaehwa Jung added a comment - - edited

        When I adjust the patch about jira issue named TAJO-284, it made a error to scan hdfs blocks. So, I thought that Scanning just one file is sure the main cause of error. But I think now that I misunderstood about the cause. It needs to improve Planner compare than StorageManager and Scanner. So, I'm going to modify the issue title.

        Show
        blrunner Jaehwa Jung added a comment - - edited When I adjust the patch about jira issue named TAJO-284 , it made a error to scan hdfs blocks. So, I thought that Scanning just one file is sure the main cause of error. But I think now that I misunderstood about the cause. It needs to improve Planner compare than StorageManager and Scanner. So, I'm going to modify the issue title.
        Hide
        hyunsik Hyunsik Choi added a comment -

        Could you explain this issue in more detail?

        Show
        hyunsik Hyunsik Choi added a comment - Could you explain this issue in more detail?

          People

          • Assignee:
            hyunsik Hyunsik Choi
            Reporter:
            blrunner Jaehwa Jung
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development