Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.0
    • Component/s: Catalog
    • Labels:
      None

      Description

      Query master need to know where partitions of memory cached table locate.
      At least we need a meta table contain such information

      partition_id
      partition_value
      ordinal_position
      locations

      Any suggestion?

      1. TAJO-475_140127_112723.patch
        248 kB
        Hyunsik Choi
      2. TAJO-475_140127_120007.patch
        258 kB
        Hyunsik Choi
      3. TAJO-475_140127_172418.patch
        259 kB
        Hyunsik Choi
      4. TAJO-475_2.patch
        218 kB
        Hyunsik Choi
      5. tajo-475.diff
        228 kB
        Min Zhou
      6. tajo-475-v2.patch
        226 kB
        Min Zhou
      7. tajo-475-v3.patch
        242 kB
        Min Zhou
      8. tajo-475-v4.patch
        227 kB
        Min Zhou

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Tajo-master-build #32 (See https://builds.apache.org/job/Tajo-master-build/32/)
          TAJO-475: Table partition catalog recap. (Min Zhou and hyunsik) (hyunsik: https://git-wip-us.apache.org/repos/asf?p=incubator-tajo.git&a=commit&h=eb563addd336019417d045b95277dec9f89b8f5f)

          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/LogicalPlan.java
          • tajo-catalog/tajo-catalog-client/src/main/java/org/apache/tajo/catalog/AbstractCatalogClient.java
          • tajo-core/tajo-core-backend/src/test/resources/results/testBuildDDL.result
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/PlannerUtil.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/parser/SQLAnalyzer.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/PersistentStoreNode.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/PhysicalPlannerImpl.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/CreateTableNode.java
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/DDLBuilder.java
          • tajo-client/src/main/java/org/apache/tajo/cli/DescTableCommand.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/rewrite/PartitionedTableRewriter.java
          • tajo-client/src/main/proto/ClientProtos.proto
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/partition/PartitionMethodDesc.java
          • tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/query/TestTablePartitions.java
          • tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/query/TestCTASQuery.java
          • tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/parser/TestHiveConverter.java
          • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/DerbyStore.java
          • tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogStore.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/InsertNode.java
          • tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestDBStore.java
          • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/MemStore.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/utils/SchemaUtil.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/StoreTableNode.java
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/partition/Specifier.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/global/GlobalPlanner.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/GlobalEngine.java
          • tajo-catalog/tajo-catalog-client/src/main/proto/CatalogProtocol.proto
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/Query.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/TajoMasterClientService.java
          • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/IndexDesc.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/ScanNode.java
          • tajo-catalog/tajo-catalog-client/src/main/java/org/apache/tajo/catalog/CatalogService.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/PartitionedTableScanNode.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/query/QueryContext.java
          • tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/client/TestTajoClient.java
          • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java
          • tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/query/TestUnionQuery.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/SeqScanExec.java
          • tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalog.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/ShuffleFileWriteNode.java
          • tajo-algebra/src/main/java/org/apache/tajo/algebra/CreateTable.java
          • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/CatalogStore.java
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogConstants.java
          • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/MySQLStore.java
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/Schema.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/LogicalPlanner.java
          • CHANGES.txt
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/partition/PartitionDesc.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/rewrite/ProjectionPushDownRule.java
          • tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/TableDesc.java
          • tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/client/TestDDLBuilder.java
          • tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/parser/TestSQLAnalyzer.java
          • tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/test/java/org/apache/tajo/catalog/store/TestHCatalogStore.java
          • tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/planner/TestLogicalPlanner.java
          • tajo-core/tajo-core-backend/pom.xml
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/ColumnPartitionedTableStoreExec.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/HashAggregateExec.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Tajo-master-build #32 (See https://builds.apache.org/job/Tajo-master-build/32/ ) TAJO-475 : Table partition catalog recap. (Min Zhou and hyunsik) (hyunsik: https://git-wip-us.apache.org/repos/asf?p=incubator-tajo.git&a=commit&h=eb563addd336019417d045b95277dec9f89b8f5f ) tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/LogicalPlan.java tajo-catalog/tajo-catalog-client/src/main/java/org/apache/tajo/catalog/AbstractCatalogClient.java tajo-core/tajo-core-backend/src/test/resources/results/testBuildDDL.result tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/PlannerUtil.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/parser/SQLAnalyzer.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/PersistentStoreNode.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/PhysicalPlannerImpl.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/CreateTableNode.java tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/DDLBuilder.java tajo-client/src/main/java/org/apache/tajo/cli/DescTableCommand.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/rewrite/PartitionedTableRewriter.java tajo-client/src/main/proto/ClientProtos.proto tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/partition/PartitionMethodDesc.java tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/query/TestTablePartitions.java tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/query/TestCTASQuery.java tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/parser/TestHiveConverter.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/DerbyStore.java tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogStore.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/InsertNode.java tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestDBStore.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/MemStore.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/utils/SchemaUtil.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/StoreTableNode.java tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/partition/Specifier.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/global/GlobalPlanner.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/GlobalEngine.java tajo-catalog/tajo-catalog-client/src/main/proto/CatalogProtocol.proto tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/Query.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/TajoMasterClientService.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/IndexDesc.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/ScanNode.java tajo-catalog/tajo-catalog-client/src/main/java/org/apache/tajo/catalog/CatalogService.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/PartitionedTableScanNode.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/query/QueryContext.java tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/client/TestTajoClient.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/query/TestUnionQuery.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/SeqScanExec.java tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalog.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/ShuffleFileWriteNode.java tajo-algebra/src/main/java/org/apache/tajo/algebra/CreateTable.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/CatalogStore.java tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogConstants.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/MySQLStore.java tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/Schema.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/LogicalPlanner.java CHANGES.txt tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/partition/PartitionDesc.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/rewrite/ProjectionPushDownRule.java tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/TableDesc.java tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/client/TestDDLBuilder.java tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/parser/TestSQLAnalyzer.java tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/test/java/org/apache/tajo/catalog/store/TestHCatalogStore.java tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/planner/TestLogicalPlanner.java tajo-core/tajo-core-backend/pom.xml tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/ColumnPartitionedTableStoreExec.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/HashAggregateExec.java
          Hide
          hyunsik Hyunsik Choi added a comment -

          committed it to master branch. Thank you for the contribution and reviews.

          Show
          hyunsik Hyunsik Choi added a comment - committed it to master branch. Thank you for the contribution and reviews.
          Hide
          jihoonson Jihoon Son added a comment -

          +1 for the latest patch. Ship it!

          Show
          jihoonson Jihoon Son added a comment - +1 for the latest patch. Ship it!
          Hide
          hyunsik Hyunsik Choi added a comment -

          This diff reflected the latest comments.

          Updated the review request against branch master in reviewboard
          https://reviews.apache.org/r/16898/

          Show
          hyunsik Hyunsik Choi added a comment - This diff reflected the latest comments. Updated the review request against branch master in reviewboard https://reviews.apache.org/r/16898/
          Hide
          jihoonson Jihoon Son added a comment -

          Min and Hyunsik, thanks for your efforts!
          I'm reviewing the latest patch, and will leave some feedback on RB.

          Show
          jihoonson Jihoon Son added a comment - Min and Hyunsik, thanks for your efforts! I'm reviewing the latest patch, and will leave some feedback on RB.
          Hide
          tajoqa Tajo QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12625311/TAJO-475_140127_120007.patch
          against master revision accd0e5.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 12 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 javadoc. The applied patch generated 48 javadoc warnings (more than the trunk's current 47 warnings).

          +1 checkstyle. The patch generated 0 code style errors.

          -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in tajo-algebra tajo-catalog/tajo-catalog-client tajo-catalog/tajo-catalog-common tajo-catalog/tajo-catalog-drivers/tajo-hcatalog tajo-catalog/tajo-catalog-server tajo-client tajo-core/tajo-core-backend:
          org.apache.tajo.engine.function.TestBuiltinFunctions
          org.apache.tajo.jdbc.TestTajoJdbc
          org.apache.tajo.engine.query.TestNullValues
          org.apache.tajo.engine.query.TestUnionQuery
          org.apache.tajo.engine.query.TestGroupByQuery
          org.apache.tajo.engine.query.TestSortQuery
          org.apache.tajo.storage.TestRowFile
          org.apache.tajo.benchmark.TestTPCH
          org.apache.tajo.engine.query.TestJoinQuery
          org.apache.tajo.engine.query.TestCaseByCases
          org.apache.tajo.client.TestTajoClient
          org.apache.tajo.engine.query.TestSelectQuery
          org.apache.tajo.engine.query.TestInsertQuery
          org.apache.tajo.engine.query.TestCreateTable
          org.apache.tajo.engine.query.TestCTASQuery
          org.apache.tajo.engine.query.TestTableSubQuery
          org.apache.tajo.engine.query.TestTablePartitions

          Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/79//testReport/
          Findbugs results: https://builds.apache.org/job/PreCommit-TAJO-Build/79//findbugsResult
          Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/79//console

          This message is automatically generated.

          Show
          tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12625311/TAJO-475_140127_120007.patch against master revision accd0e5. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 12 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 javadoc. The applied patch generated 48 javadoc warnings (more than the trunk's current 47 warnings). +1 checkstyle. The patch generated 0 code style errors. -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in tajo-algebra tajo-catalog/tajo-catalog-client tajo-catalog/tajo-catalog-common tajo-catalog/tajo-catalog-drivers/tajo-hcatalog tajo-catalog/tajo-catalog-server tajo-client tajo-core/tajo-core-backend: org.apache.tajo.engine.function.TestBuiltinFunctions org.apache.tajo.jdbc.TestTajoJdbc org.apache.tajo.engine.query.TestNullValues org.apache.tajo.engine.query.TestUnionQuery org.apache.tajo.engine.query.TestGroupByQuery org.apache.tajo.engine.query.TestSortQuery org.apache.tajo.storage.TestRowFile org.apache.tajo.benchmark.TestTPCH org.apache.tajo.engine.query.TestJoinQuery org.apache.tajo.engine.query.TestCaseByCases org.apache.tajo.client.TestTajoClient org.apache.tajo.engine.query.TestSelectQuery org.apache.tajo.engine.query.TestInsertQuery org.apache.tajo.engine.query.TestCreateTable org.apache.tajo.engine.query.TestCTASQuery org.apache.tajo.engine.query.TestTableSubQuery org.apache.tajo.engine.query.TestTablePartitions Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/79//testReport/ Findbugs results: https://builds.apache.org/job/PreCommit-TAJO-Build/79//findbugsResult Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/79//console This message is automatically generated.
          Hide
          hyunsik Hyunsik Choi added a comment -

          fixed the build errors of hcatalog driver

          Updated the review request against branch master in reviewboard
          https://reviews.apache.org/r/16898/

          Show
          hyunsik Hyunsik Choi added a comment - fixed the build errors of hcatalog driver Updated the review request against branch master in reviewboard https://reviews.apache.org/r/16898/
          Hide
          hyunsik Hyunsik Choi added a comment -

          The updated patch additionally does as follows:

          • Add getLogicalSchema and getPhysicalSchema to ScanNode
            • getLogicalSchema provides a schema including partition columns, and it is used for logical planning
            • getPhysicalSchema just provides a schema for stored columns, and it will be used for actual scanners
          • Add logical schema to TableDesc
          • Add TableSchema to RelationNode
          • Improve SeqScanNode and rewrite method to use logical schema
          • Simplify and cleanup the succeeded event in Query

          The main objective of this additional work is to separate the schema of ScanNode into more clear definitions. So far, we ambiguously use an output schema as a table schema of ScanNode in many cases. It have made us much confuse. This work fixes this problem.

          For example, consider a following schema with partition description:

          create table tb1 (col1 int4, col2 int4) partition by column(key float8) 
          

          Its logical schema will be (col1 int4, col2 int4, key float8), whereas its physical schema will be (col1 int4, col2 int4). The output schema of ScanNode is just determined by a target list of ScanNode.

          Show
          hyunsik Hyunsik Choi added a comment - The updated patch additionally does as follows: Add getLogicalSchema and getPhysicalSchema to ScanNode getLogicalSchema provides a schema including partition columns, and it is used for logical planning getPhysicalSchema just provides a schema for stored columns, and it will be used for actual scanners Add logical schema to TableDesc Add TableSchema to RelationNode Improve SeqScanNode and rewrite method to use logical schema Simplify and cleanup the succeeded event in Query The main objective of this additional work is to separate the schema of ScanNode into more clear definitions. So far, we ambiguously use an output schema as a table schema of ScanNode in many cases. It have made us much confuse. This work fixes this problem. For example, consider a following schema with partition description: create table tb1 (col1 int4, col2 int4) partition by column(key float8) Its logical schema will be (col1 int4, col2 int4, key float8), whereas its physical schema will be (col1 int4, col2 int4). The output schema of ScanNode is just determined by a target list of ScanNode.
          Hide
          tajoqa Tajo QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12625305/TAJO-475_140127_112723.patch
          against master revision accd0e5.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 11 new or modified test files.

          -1 javac. The patch appears to cause the build to fail.

          Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/78//console

          This message is automatically generated.

          Show
          tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12625305/TAJO-475_140127_112723.patch against master revision accd0e5. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 11 new or modified test files. -1 javac. The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/78//console This message is automatically generated.
          Hide
          hyunsik Hyunsik Choi added a comment -

          Updated the review request against branch master in reviewboard
          https://reviews.apache.org/r/16898/

          Show
          hyunsik Hyunsik Choi added a comment - Updated the review request against branch master in reviewboard https://reviews.apache.org/r/16898/
          Hide
          hyunsik Hyunsik Choi added a comment -

          Min worked on this issue. Recently, he has worked on several issues, such as TAJO-521, TAJO-522, and TAJO-525. Due to his other schedules and the release due, we decided that I take this issues and continue.

          Show
          hyunsik Hyunsik Choi added a comment - Min worked on this issue. Recently, he has worked on several issues, such as TAJO-521 , TAJO-522 , and TAJO-525 . Due to his other schedules and the release due, we decided that I take this issues and continue.
          Hide
          coderplay Min Zhou added a comment -

          Hyunsik Choi
          Really appreciate your job on this.

          Actually, I am fighting with TAJO-521, still can't make the real cluster work fine.

          Show
          coderplay Min Zhou added a comment - Hyunsik Choi Really appreciate your job on this. Actually, I am fighting with TAJO-521 , still can't make the real cluster work fine.
          Hide
          tajoqa Tajo QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12623857/TAJO-475_2.patch
          against master revision ebb12b7.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 8 new or modified test files.

          -1 javac. The patch appears to cause the build to fail.

          Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/60//console

          This message is automatically generated.

          Show
          tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12623857/TAJO-475_2.patch against master revision ebb12b7. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 8 new or modified test files. -1 javac. The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/60//console This message is automatically generated.
          Hide
          hyunsik Hyunsik Choi added a comment -

          In TAJO-501, there ware many changes in LogicalPlanner. So, I rebased this patch against the latest revision. I hope that it would be helpful to you.

          Show
          hyunsik Hyunsik Choi added a comment - In TAJO-501 , there ware many changes in LogicalPlanner. So, I rebased this patch against the latest revision. I hope that it would be helpful to you.
          Hide
          tajoqa Tajo QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12623062/tajo-475-v4.patch
          against master revision 69700bf.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 5 new or modified test files.

          -1 javac. The patch appears to cause the build to fail.

          Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/42//console

          This message is automatically generated.

          Show
          tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12623062/tajo-475-v4.patch against master revision 69700bf. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 5 new or modified test files. -1 javac. The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/42//console This message is automatically generated.
          Hide
          jihoonson Jihoon Son added a comment -

          Thanks, Min.
          The patch passed every test successfully.
          I'll review after about 5 hours.

          Show
          jihoonson Jihoon Son added a comment - Thanks, Min. The patch passed every test successfully. I'll review after about 5 hours.
          Hide
          coderplay Min Zhou added a comment -

          Hyunsik Choi Jihoon Son

          Please notice that I add a verification in LogicalPlanVerifier.java. Previously user can create partitioned table like this way

          CREATE TABLE (foo int, bar string) PARTITION BY COLUMN( another_col int);
          

          This verifier will forbid that kind of sql, because as an offline discussion with Hyunsik. The partitioned condition will support expression as well. Furthermore, only column partitioned table support virtual column. If we continue to support hash/range/list partition, I think we should add this verification.

          Show
          coderplay Min Zhou added a comment - Hyunsik Choi Jihoon Son Please notice that I add a verification in LogicalPlanVerifier.java. Previously user can create partitioned table like this way CREATE TABLE (foo int, bar string) PARTITION BY COLUMN( another_col int); This verifier will forbid that kind of sql, because as an offline discussion with Hyunsik. The partitioned condition will support expression as well. Furthermore, only column partitioned table support virtual column. If we continue to support hash/range/list partition, I think we should add this verification.
          Hide
          coderplay Min Zhou added a comment -

          Submitted a wrong patch before( the v3)

          Show
          coderplay Min Zhou added a comment - Submitted a wrong patch before( the v3)
          Hide
          hyunsik Hyunsik Choi added a comment -

          It's awesome work! I'm happy to review this work.

          BTW, the latest diff includes other revisions diff too as you can see the below:
          https://reviews.apache.org/r/16898/diff/3/#0.1

          You may create the patch against some old revision. Could you check this?

          Show
          hyunsik Hyunsik Choi added a comment - It's awesome work! I'm happy to review this work. BTW, the latest diff includes other revisions diff too as you can see the below: https://reviews.apache.org/r/16898/diff/3/#0.1 You may create the patch against some old revision. Could you check this?
          Hide
          coderplay Min Zhou added a comment -

          Resubmit a patch , catching up with the pace of the newest revision.

          Show
          coderplay Min Zhou added a comment - Resubmit a patch , catching up with the pace of the newest revision.
          Hide
          coderplay Min Zhou added a comment -

          Done. Thank you

          Show
          coderplay Min Zhou added a comment - Done. Thank you
          Hide
          jihoonson Jihoon Son added a comment -

          Thanks, Min
          but the patch looks to have gone stale. Would you update the patch again after rebasing?

          Show
          jihoonson Jihoon Son added a comment - Thanks, Min but the patch looks to have gone stale. Would you update the patch again after rebasing?
          Hide
          coderplay Min Zhou added a comment -

          Can you at first apply that patch, and see if the test cases can pass there?

          Show
          coderplay Min Zhou added a comment - Can you at first apply that patch, and see if the test cases can pass there?
          Hide
          coderplay Min Zhou added a comment -

          Ok, I already filed a review request on https://reviews.apache.org/r/16898

          Show
          coderplay Min Zhou added a comment - Ok, I already filed a review request on https://reviews.apache.org/r/16898
          Hide
          jihoonson Jihoon Son added a comment -

          Great work!
          As you said, the patch is big.
          If you upload the patch at RB, it would be great for us to review it.

          Thanks.

          Show
          jihoonson Jihoon Son added a comment - Great work! As you said, the patch is big. If you upload the patch at RB, it would be great for us to review it. Thanks.
          Hide
          coderplay Min Zhou added a comment -

          This is a big patch, all test cases can go through. But that's a false image. see TAJO-505, this patch throws NPEs due to that bug, but tests can pass .. weird

          Some of the changes don't directly related to this recap, but triggers some of the bug of tajo. If I didn't fix those bugs, this patch will fail in the test cases.

          Show
          coderplay Min Zhou added a comment - This is a big patch, all test cases can go through. But that's a false image. see TAJO-505 , this patch throws NPEs due to that bug, but tests can pass .. weird Some of the changes don't directly related to this recap, but triggers some of the bug of tajo. If I didn't fix those bugs, this patch will fail in the test cases.
          Hide
          hyunsik Hyunsik Choi added a comment -

          the schema design looks nice. In addition, I'm very interested in mmap approach. That makes sense and looks very promising.

          Show
          hyunsik Hyunsik Choi added a comment - the schema design looks nice. In addition, I'm very interested in mmap approach. That makes sense and looks very promising.
          Hide
          jihoonson Jihoon Son added a comment -

          +1.
          It looks enough to represent the necessary partition information.

          Show
          jihoonson Jihoon Son added a comment - +1. It looks enough to represent the necessary partition information.
          Hide
          coderplay Min Zhou added a comment - - edited

          Here is the proto after discuss with Hyunsik offline

          message PartitionMethodProto {
            required string tableId = 1;
            required PartitionType partitionType = 2;
            required string expression = 3;
            required SchemaProto expressionSchema = 4;
          }
          
          message PartitionDescProto {
            required string tableId = 1;
            required int32 partitionMethodId = 2;
            optional string partitionName = 3;
            required int32  ordinalPosition = 4;
            optional string paritionValue = 5;
            optional string path = 6;
            optional string cacheNodes = 7;
          }
          

          At most one partiton method for one table. PARTITION_METHODS metadata records the partition type, partition expression(or columns) and the schema for that expression.

          PARTITIONS metadata records the detail of each partition, such as partition name if it has, partition value is the parition method is range/column partitioning, and hdfs path , local cache nodes path.

          If we create a tajo table like this way

           CREATE TABLE IF NOT EXISTS `user` (  
           ->   `id` int(11) NOT NULL AUTO_INCREMENT COMMENT 'user ID',  
           ->   `name` varchar(50) NOT NULL DEFAULT '' COMMENT user name',  
           ->   `sex` int(1) NOT NULL DEFAULT '0' COMMENT '0 for male,1 for female',  
           ->   PRIMARY KEY (`id`)  
           -> ) ENGINE=MyISAM  DEFAULT CHARSET=utf8 AUTO_INCREMENT=1  
           -> PARTITION BY RANGE (id) (  
           ->     PARTITION p0 VALUES LESS THAN (3),  
           ->     PARTITION p1 VALUES LESS THAN (6),  
           ->     PARTITION p2 VALUES LESS THAN (9),  
           ->     PARTITION p3 VALUES LESS THAN (12),  
           ->     PARTITION p4 VALUES LESS THAN MAXVALUE  
           -> );  
          

          The PARTITION_METHODS table in metadata will add one row, and PARTITIONS table will add 5 rows for 5 partitions.

          Show
          coderplay Min Zhou added a comment - - edited Here is the proto after discuss with Hyunsik offline message PartitionMethodProto { required string tableId = 1; required PartitionType partitionType = 2; required string expression = 3; required SchemaProto expressionSchema = 4; } message PartitionDescProto { required string tableId = 1; required int32 partitionMethodId = 2; optional string partitionName = 3; required int32 ordinalPosition = 4; optional string paritionValue = 5; optional string path = 6; optional string cacheNodes = 7; } At most one partiton method for one table. PARTITION_METHODS metadata records the partition type, partition expression(or columns) and the schema for that expression. PARTITIONS metadata records the detail of each partition, such as partition name if it has, partition value is the parition method is range/column partitioning, and hdfs path , local cache nodes path. If we create a tajo table like this way CREATE TABLE IF NOT EXISTS `user` ( -> `id` int(11) NOT NULL AUTO_INCREMENT COMMENT 'user ID', -> `name` varchar(50) NOT NULL DEFAULT '' COMMENT user name', -> `sex` int(1) NOT NULL DEFAULT '0' COMMENT '0 for male,1 for female', -> PRIMARY KEY (`id`) -> ) ENGINE=MyISAM DEFAULT CHARSET=utf8 AUTO_INCREMENT=1 -> PARTITION BY RANGE (id) ( -> PARTITION p0 VALUES LESS THAN (3), -> PARTITION p1 VALUES LESS THAN (6), -> PARTITION p2 VALUES LESS THAN (9), -> PARTITION p3 VALUES LESS THAN (12), -> PARTITION p4 VALUES LESS THAN MAXVALUE -> ); The PARTITION_METHODS table in metadata will add one row, and PARTITIONS table will add 5 rows for 5 partitions.
          Hide
          coderplay Min Zhou added a comment - - edited

          Yes, your correctly understood my meaning. For the first step, we can build a pure in-memory data structure. That's quite fast and straightforward. However, for a long term goal. We should think about those aspect.

          1. Memory is expensive. There is a proverb, use our limited funds where they can be put to best use. We can cut down the footprint through compression, LRU cache and multiply-layer storage memory ->SSD -> hard disk.

          2. I am not familiar with the yarn mode of tajo. From my knowledge, the worker is spawned by nodemanager on demand. Since the workers can't always standup, they can't keep data in-memory for sharing with subsequent queries. A solution is put this cache manager as a aux service in nodemanager like shuffle service in hadoop mapreduce.

          3. If we embed a cache service in nodemanager, we should think about how the workers can share cached data from that service. That is an inter-processes sharing. The most efficient way to achieve on unix system is mmap. see http://en.wikipedia.org/wiki/Mmap . In java, Filechannel can mapping a file into memory, its backend is mmap. To be better, if we use those kind of direct memory rather than allocated from heap, we can get a zero GC overhead. see https://gist.github.com/coderplay/8262536 for example.

          Thanks for the feedback, really encouraging.

          Min

          Show
          coderplay Min Zhou added a comment - - edited Yes, your correctly understood my meaning. For the first step, we can build a pure in-memory data structure. That's quite fast and straightforward. However, for a long term goal. We should think about those aspect. 1. Memory is expensive. There is a proverb, use our limited funds where they can be put to best use. We can cut down the footprint through compression, LRU cache and multiply-layer storage memory ->SSD -> hard disk. 2. I am not familiar with the yarn mode of tajo. From my knowledge, the worker is spawned by nodemanager on demand. Since the workers can't always standup, they can't keep data in-memory for sharing with subsequent queries. A solution is put this cache manager as a aux service in nodemanager like shuffle service in hadoop mapreduce. 3. If we embed a cache service in nodemanager, we should think about how the workers can share cached data from that service. That is an inter-processes sharing. The most efficient way to achieve on unix system is mmap. see http://en.wikipedia.org/wiki/Mmap . In java, Filechannel can mapping a file into memory, its backend is mmap. To be better, if we use those kind of direct memory rather than allocated from heap, we can get a zero GC overhead. see https://gist.github.com/coderplay/8262536 for example. Thanks for the feedback, really encouraging. Min
          Hide
          hyunsik Hyunsik Choi added a comment -

          I misunderstood memory-cached table as in-memory data structure for partitions. Anyway, your proposal makes sense. I'm looking forward to seeing this recap.

          Show
          hyunsik Hyunsik Choi added a comment - I misunderstood memory-cached table as in-memory data structure for partitions. Anyway, your proposal makes sense. I'm looking forward to seeing this recap.
          Hide
          hyunsik Hyunsik Choi added a comment -

          +1 for the proposed idea.

          Yes, we should make use of the partition pruning with RDBMS index. In addition, as you mentioned, in-memory data structures for partitions will be very useful and the ultimate one that we have to go.

          Show
          hyunsik Hyunsik Choi added a comment - +1 for the proposed idea. Yes, we should make use of the partition pruning with RDBMS index. In addition, as you mentioned, in-memory data structures for partitions will be very useful and the ultimate one that we have to go.
          Hide
          coderplay Min Zhou added a comment -

          Here is the meta table for paritions in mysql, I perfer this design. They don't store the number of partitions, just each partition has one row.

          mysql> desc PARTITIONS;
          +-------------------------------+---------------------+------+-----+---------+-------+
          | Field                         | Type                | Null | Key | Default | Extra |
          +-------------------------------+---------------------+------+-----+---------+-------+
          | TABLE_CATALOG                 | varchar(512)        | YES  |     | NULL    |       |
          | TABLE_SCHEMA                  | varchar(64)         | NO   |     |         |       |
          | TABLE_NAME                    | varchar(64)         | NO   |     |         |       |
          | PARTITION_NAME                | varchar(64)         | YES  |     | NULL    |       |
          | SUBPARTITION_NAME             | varchar(64)         | YES  |     | NULL    |       |
          | PARTITION_ORDINAL_POSITION    | bigint(21) unsigned | YES  |     | NULL    |       |
          | SUBPARTITION_ORDINAL_POSITION | bigint(21) unsigned | YES  |     | NULL    |       |
          | PARTITION_METHOD              | varchar(12)         | YES  |     | NULL    |       |
          | SUBPARTITION_METHOD           | varchar(12)         | YES  |     | NULL    |       |
          | PARTITION_EXPRESSION          | longtext            | YES  |     | NULL    |       |
          | SUBPARTITION_EXPRESSION       | longtext            | YES  |     | NULL    |       |
          | PARTITION_DESCRIPTION         | longtext            | YES  |     | NULL    |       |
          | TABLE_ROWS                    | bigint(21) unsigned | NO   |     | 0       |       |
          | AVG_ROW_LENGTH                | bigint(21) unsigned | NO   |     | 0       |       |
          | DATA_LENGTH                   | bigint(21) unsigned | NO   |     | 0       |       |
          | MAX_DATA_LENGTH               | bigint(21) unsigned | YES  |     | NULL    |       |
          | INDEX_LENGTH                  | bigint(21) unsigned | NO   |     | 0       |       |
          | DATA_FREE                     | bigint(21) unsigned | NO   |     | 0       |       |
          | CREATE_TIME                   | datetime            | YES  |     | NULL    |       |
          | UPDATE_TIME                   | datetime            | YES  |     | NULL    |       |
          | CHECK_TIME                    | datetime            | YES  |     | NULL    |       |
          | CHECKSUM                      | bigint(21) unsigned | YES  |     | NULL    |       |
          | PARTITION_COMMENT             | varchar(80)         | NO   |     |         |       |
          | NODEGROUP                     | varchar(12)         | NO   |     |         |       |
          | TABLESPACE_NAME               | varchar(64)         | YES  |     | NULL    |       |
          +-------------------------------+---------------------+------+-----+---------+-------+
          

          If I create a partitioned table in mysql like this way

          mysql> CREATE TABLE IF NOT EXISTS `user` (  
           ->   `id` int(11) NOT NULL AUTO_INCREMENT COMMENT 'user ID',  
           ->   `name` varchar(50) NOT NULL DEFAULT '' COMMENT user name',  
           ->   `sex` int(1) NOT NULL DEFAULT '0' COMMENT '0 for male,1 for female',  
           ->   PRIMARY KEY (`id`)  
           -> ) ENGINE=MyISAM  DEFAULT CHARSET=utf8 AUTO_INCREMENT=1  
           -> PARTITION BY RANGE (id) (  
           ->     PARTITION p0 VALUES LESS THAN (3),  
           ->     PARTITION p1 VALUES LESS THAN (6),  
           ->     PARTITION p2 VALUES LESS THAN (9),  
           ->     PARTITION p3 VALUES LESS THAN (12),  
           ->     PARTITION p4 VALUES LESS THAN MAXVALUE  
           -> );  
          

          The metadata will be like this

          mysql> select TABLE_NAME, PARTITION_NAME, PARTITION_ORDINAL_POSITION, PARTITION_METHOD, PARTITION_EXPRESSION, PARTITION_DESCRIPTION, NODEGROUP from PARTITIONS where TABLE_NAME='user';
          +------------+----------------+----------------------------+------------------+----------------------+-----------------------+-----------+
          | TABLE_NAME | PARTITION_NAME | PARTITION_ORDINAL_POSITION | PARTITION_METHOD | PARTITION_EXPRESSION | PARTITION_DESCRIPTION | NODEGROUP |
          +------------+----------------+----------------------------+------------------+----------------------+-----------------------+-----------+
          | user       | p0             |                          1 | RANGE            | id                   | 3                     | default   |
          | user       | p1             |                          2 | RANGE            | id                   | 6                     | default   |
          | user       | p2             |                          3 | RANGE            | id                   | 9                     | default   |
          | user       | p3             |                          4 | RANGE            | id                   | 12                    | default   |
          | user       | p4             |                          5 | RANGE            | id                   | MAXVALUE              | default   |
          +------------+----------------+----------------------------+------------------+----------------------+-----------------------+-----------+
          

          This is quite good for partition pruning and memory cached table .

          Does it make sense?

          Show
          coderplay Min Zhou added a comment - Here is the meta table for paritions in mysql, I perfer this design. They don't store the number of partitions, just each partition has one row. mysql> desc PARTITIONS; +-------------------------------+---------------------+------+-----+---------+-------+ | Field | Type | Null | Key | Default | Extra | +-------------------------------+---------------------+------+-----+---------+-------+ | TABLE_CATALOG | varchar(512) | YES | | NULL | | | TABLE_SCHEMA | varchar(64) | NO | | | | | TABLE_NAME | varchar(64) | NO | | | | | PARTITION_NAME | varchar(64) | YES | | NULL | | | SUBPARTITION_NAME | varchar(64) | YES | | NULL | | | PARTITION_ORDINAL_POSITION | bigint(21) unsigned | YES | | NULL | | | SUBPARTITION_ORDINAL_POSITION | bigint(21) unsigned | YES | | NULL | | | PARTITION_METHOD | varchar(12) | YES | | NULL | | | SUBPARTITION_METHOD | varchar(12) | YES | | NULL | | | PARTITION_EXPRESSION | longtext | YES | | NULL | | | SUBPARTITION_EXPRESSION | longtext | YES | | NULL | | | PARTITION_DESCRIPTION | longtext | YES | | NULL | | | TABLE_ROWS | bigint(21) unsigned | NO | | 0 | | | AVG_ROW_LENGTH | bigint(21) unsigned | NO | | 0 | | | DATA_LENGTH | bigint(21) unsigned | NO | | 0 | | | MAX_DATA_LENGTH | bigint(21) unsigned | YES | | NULL | | | INDEX_LENGTH | bigint(21) unsigned | NO | | 0 | | | DATA_FREE | bigint(21) unsigned | NO | | 0 | | | CREATE_TIME | datetime | YES | | NULL | | | UPDATE_TIME | datetime | YES | | NULL | | | CHECK_TIME | datetime | YES | | NULL | | | CHECKSUM | bigint(21) unsigned | YES | | NULL | | | PARTITION_COMMENT | varchar(80) | NO | | | | | NODEGROUP | varchar(12) | NO | | | | | TABLESPACE_NAME | varchar(64) | YES | | NULL | | +-------------------------------+---------------------+------+-----+---------+-------+ If I create a partitioned table in mysql like this way mysql> CREATE TABLE IF NOT EXISTS `user` ( -> `id` int(11) NOT NULL AUTO_INCREMENT COMMENT 'user ID', -> `name` varchar(50) NOT NULL DEFAULT '' COMMENT user name', -> `sex` int(1) NOT NULL DEFAULT '0' COMMENT '0 for male,1 for female', -> PRIMARY KEY (`id`) -> ) ENGINE=MyISAM DEFAULT CHARSET=utf8 AUTO_INCREMENT=1 -> PARTITION BY RANGE (id) ( -> PARTITION p0 VALUES LESS THAN (3), -> PARTITION p1 VALUES LESS THAN (6), -> PARTITION p2 VALUES LESS THAN (9), -> PARTITION p3 VALUES LESS THAN (12), -> PARTITION p4 VALUES LESS THAN MAXVALUE -> ); The metadata will be like this mysql> select TABLE_NAME, PARTITION_NAME, PARTITION_ORDINAL_POSITION, PARTITION_METHOD, PARTITION_EXPRESSION, PARTITION_DESCRIPTION, NODEGROUP from PARTITIONS where TABLE_NAME='user'; +------------+----------------+----------------------------+------------------+----------------------+-----------------------+-----------+ | TABLE_NAME | PARTITION_NAME | PARTITION_ORDINAL_POSITION | PARTITION_METHOD | PARTITION_EXPRESSION | PARTITION_DESCRIPTION | NODEGROUP | +------------+----------------+----------------------------+------------------+----------------------+-----------------------+-----------+ | user | p0 | 1 | RANGE | id | 3 | default | | user | p1 | 2 | RANGE | id | 6 | default | | user | p2 | 3 | RANGE | id | 9 | default | | user | p3 | 4 | RANGE | id | 12 | default | | user | p4 | 5 | RANGE | id | MAXVALUE | default | +------------+----------------+----------------------------+------------------+----------------------+-----------------------+-----------+ This is quite good for partition pruning and memory cached table . Does it make sense?

            People

            • Assignee:
              coderplay Min Zhou
              Reporter:
              coderplay Min Zhou
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development