Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.12.0
    • Component/s: Catalog, Planner/Optimizer
    • Labels:
      None

      Description

      Currently, PartitionedTableRewriter take a look into partition directories for rewriting filter conditions. It get all sub directories of table path because catalog doesn’t provide partition directories. But if there are lots of sub directories on HDFS, such as, more than 10,000 directories, it might be cause overload to NameNode. Thus, CatalogStore need to provide partition directories for specified filter conditions. I designed new method to CatalogStore as follows:

      • method name: getPartitionsWithConditionFilters
      • first parameter: database name
      • second parameter: table name
      • third parameter: where clause (included target column name and partition value)
      • return values: List<org.apache.tajo.catalog.proto.CatalogProtos.TablePartitionProto>
      • description: It scan right partition directories on CatalogStore with where caluse.
        For examples, users set parameters as following:
        • first parameter: default
        • second parameter: table1
        • third parameter: COLUMN_NAME = 'col1' AND PARTITION_VALUE = '3

      In the previous cases, this method will create select clause as follows.

      SELECT DISTINCT A.PATH
      FROM PARTITIONS A, (
        SELECT B.PARTITION_ID
        FROM PARTITION_KEYS B
        WHERE B.PARTITION_ID > 0 
        AND (
          COLUMN_NAME = 'col1' AND PARTITION_VALUE = '3'
        )
      ) B
      WHERE A.PARTITION_ID > 0
      AND A.TID = ${table_id}
      AND A.PARTITION_ID = B.PARTITION_ID
      

      At the first time, I considered to use EvalNode instead of where clause. But I can’t use it because of recursive related problems between tajo-catalog module and tajo-plan module. So, I’ll implement utility class to convert EvalNode to SQL.

      1. TAJO-1493_2.patch
        137 kB
        Jaehwa Jung
      2. TAJO-1493_3.patch
        138 kB
        Jaehwa Jung
      3. TAJO-1493_4.patch
        142 kB
        Jaehwa Jung
      4. TAJO-1493_5.patch
        172 kB
        Jaehwa Jung
      5. TAJO-1493_6.patch
        182 kB
        Jaehwa Jung
      6. TAJO-1493.patch
        133 kB
        Jaehwa Jung

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user blrunner opened a pull request:

          https://github.com/apache/tajo/pull/624

          TAJO-1493: Add a method to get partition directories with filter conditions.

          Not yet implemented unit test cases.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/blrunner/tajo TAJO-1493

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/tajo/pull/624.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #624


          commit 743122bade809e5a6fd6ccbb5025a87233e0ac40
          Author: JaeHwa Jung <blrunner@apache.org>
          Date: 2015-07-06T01:49:20Z

          TAJO-1493: Add a method to get partition directories with filter conditions.


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user blrunner opened a pull request: https://github.com/apache/tajo/pull/624 TAJO-1493 : Add a method to get partition directories with filter conditions. Not yet implemented unit test cases. You can merge this pull request into a Git repository by running: $ git pull https://github.com/blrunner/tajo TAJO-1493 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/tajo/pull/624.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #624 commit 743122bade809e5a6fd6ccbb5025a87233e0ac40 Author: JaeHwa Jung <blrunner@apache.org> Date: 2015-07-06T01:49:20Z TAJO-1493 : Add a method to get partition directories with filter conditions.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user blrunner commented on the pull request:

          https://github.com/apache/tajo/pull/624#issuecomment-127042279

          I implemented CatalogStore::getPartitionsByDirectSql which execute a select statement to CatalogStore for searching right partition directories. But current tajo testing cluster use MemStore for default CatalogStore and I couldn't execute the select statement because MemStore just retain all catalog informations to HashMap. If you want to verify this patch, you can test as following:

          ```
          mvn clean install -Pparallel-test -DLOG_LEVEL=WARN -Dmaven.fork.count=2 -Dtajo.catalog.store.class=org.apache.tajo.catalog.store.DerbyStore
          ```

          Show
          githubbot ASF GitHub Bot added a comment - Github user blrunner commented on the pull request: https://github.com/apache/tajo/pull/624#issuecomment-127042279 I implemented CatalogStore::getPartitionsByDirectSql which execute a select statement to CatalogStore for searching right partition directories. But current tajo testing cluster use MemStore for default CatalogStore and I couldn't execute the select statement because MemStore just retain all catalog informations to HashMap. If you want to verify this patch, you can test as following: ``` mvn clean install -Pparallel-test -DLOG_LEVEL=WARN -Dmaven.fork.count=2 -Dtajo.catalog.store.class=org.apache.tajo.catalog.store.DerbyStore ```
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user blrunner commented on the pull request:

          https://github.com/apache/tajo/pull/624#issuecomment-127275276

          I found that this patch run as expected with HiveCatalogStore and MySQLStore on my testing cluster. And simple query response had been reported as following:

          • Table schema:
            ```
            create table partitioned_lineitem (L_SUPPKEY bigint, L_LINENUMBER bigint,
            L_QUANTITY double, L_EXTENDEDPRICE double, L_DISCOUNT double, L_TAX double, L_RETURNFLAG text, L_LINESTATUS text,
            L_SHIPDATE text, L_COMMITDATE text, L_RECEIPTDATE text, L_SHIPINSTRUCT text, L_SHIPMODE text, L_COMMENT text)
            partition by column (L_ORDERKEY bigint, L_PARTKEY bigint)
            ```
          • Partition numbers: 100,000
          • Select statement: select * from partitioned_lineitem limit 10;
          • Response time:
          • previous rewriter: 15 ~ 16 sec
          • improved rewriter: 12 ~ 13 sec

          Honestly, I didn't implement unit test cases for executing queries because current almost tajo unit cases operate on MemStore. If we apply DerbyStore to some unit test cases for physical operator, we would make a lot of effort. It seems not to be the scope of this patch. So, I just added unit test cases for verifying direct sql. But if you want to test this patch with build commands, you can test with `-Dtajo.catalog.store.class` parameter as following:
          ```
          mvn clean install -Pparallel-test -DLOG_LEVEL=WARN -Dmaven.fork.count=2 -Dtajo.catalog.store.class=org.apache.tajo.catalog.store.DerbyStore
          ```

          Show
          githubbot ASF GitHub Bot added a comment - Github user blrunner commented on the pull request: https://github.com/apache/tajo/pull/624#issuecomment-127275276 I found that this patch run as expected with HiveCatalogStore and MySQLStore on my testing cluster. And simple query response had been reported as following: Table schema: ``` create table partitioned_lineitem (L_SUPPKEY bigint, L_LINENUMBER bigint, L_QUANTITY double, L_EXTENDEDPRICE double, L_DISCOUNT double, L_TAX double, L_RETURNFLAG text, L_LINESTATUS text, L_SHIPDATE text, L_COMMITDATE text, L_RECEIPTDATE text, L_SHIPINSTRUCT text, L_SHIPMODE text, L_COMMENT text) partition by column (L_ORDERKEY bigint, L_PARTKEY bigint) ``` Partition numbers: 100,000 Select statement: select * from partitioned_lineitem limit 10; Response time: previous rewriter: 15 ~ 16 sec improved rewriter: 12 ~ 13 sec Honestly, I didn't implement unit test cases for executing queries because current almost tajo unit cases operate on MemStore. If we apply DerbyStore to some unit test cases for physical operator, we would make a lot of effort. It seems not to be the scope of this patch. So, I just added unit test cases for verifying direct sql. But if you want to test this patch with build commands, you can test with `-Dtajo.catalog.store.class` parameter as following: ``` mvn clean install -Pparallel-test -DLOG_LEVEL=WARN -Dmaven.fork.count=2 -Dtajo.catalog.store.class=org.apache.tajo.catalog.store.DerbyStore ```
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/624#discussion_r36275539

          — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/MemStore.java —
          @@ -28,7 +28,15 @@
          import org.apache.tajo.catalog.*;
          import org.apache.tajo.catalog.exception.*;
          import org.apache.tajo.catalog.proto.CatalogProtos;
          -import org.apache.tajo.catalog.proto.CatalogProtos.*;
          +//import org.apache.tajo.catalog.proto.CatalogProtos.ColumnProto;
          — End diff –

          You may forget the removal of commented out lines.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/624#discussion_r36275539 — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/MemStore.java — @@ -28,7 +28,15 @@ import org.apache.tajo.catalog.*; import org.apache.tajo.catalog.exception.*; import org.apache.tajo.catalog.proto.CatalogProtos; -import org.apache.tajo.catalog.proto.CatalogProtos.*; +//import org.apache.tajo.catalog.proto.CatalogProtos.ColumnProto; — End diff – You may forget the removal of commented out lines.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/624#discussion_r36580037

          — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java —
          @@ -1195,6 +1249,71 @@ public GetTablePartitionsResponse getAllPartitions(RpcController controller, Nul
          }

          @Override
          + public GetTablePartitionsResponse getPartitionsByDirectSql(RpcController controller,
          — End diff –

          Catalog API is public APIs. In terms of this point, this API design does not make sense due to the following problems:

          • Users should know its schema and underlying persistent storage.
          • So, users should make SQL statements for all persistent storages.
          • This approach is very prone to SQL injection.
          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/624#discussion_r36580037 — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java — @@ -1195,6 +1249,71 @@ public GetTablePartitionsResponse getAllPartitions(RpcController controller, Nul } @Override + public GetTablePartitionsResponse getPartitionsByDirectSql(RpcController controller, — End diff – Catalog API is public APIs. In terms of this point, this API design does not make sense due to the following problems: Users should know its schema and underlying persistent storage. So, users should make SQL statements for all persistent storages. This approach is very prone to SQL injection.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/624#discussion_r36580081

          — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java —
          @@ -2114,6 +2114,127 @@ private void setPartitionKeys(int pid, PartitionDescProto.Builder partitionDesc)
          }

          @Override
          + public boolean existPartitions(String databaseName, String tableName) throws CatalogException {
          + Connection conn = null;
          + ResultSet res = null;
          + PreparedStatement pstmt = null;
          + boolean result = false;
          +
          + try {
          + String sql = "SELECT COUNT CNT FROM "
          + + TB_PARTTIONS +" WHERE " + COL_TABLES_PK + " = ? ";
          +
          + if (LOG.isDebugEnabled())

          { + LOG.debug(sql); + }

          +
          + int databaseId = getDatabaseId(databaseName);
          + int tableId = getTableId(databaseId, databaseName, tableName);
          +
          + conn = getConnection();
          + pstmt = conn.prepareStatement(sql);
          + pstmt.setInt(1, tableId);
          + res = pstmt.executeQuery();
          +
          + if (res.next()) {
          + if (res.getInt("CNT") > 0)

          { + result = true; + }

          + }
          + } catch (SQLException se)

          { + throw new TajoInternalError(se); + }

          finally

          { + CatalogUtil.closeQuietly(pstmt, res); + }

          + return result;
          + }
          +
          + @Override
          + public List<TablePartitionProto> getPartitionsByDirectSql(GetPartitionsWithDirectSQLRequest request)
          — End diff –

          Catalog API is public. In terms of this point, this API design does not make sense due to the following problems:

          • Users should know table schemas, their relationship, which underlying persistent storage.
          • Users should make SQL statements for all persistent storages.
          • This approach is very prone to SQL injection.

          The solution is that you can use some intermediate representation, using string or java objects. They should be transformed into SQL statements depending on persistent storages in each catalog driver.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/624#discussion_r36580081 — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java — @@ -2114,6 +2114,127 @@ private void setPartitionKeys(int pid, PartitionDescProto.Builder partitionDesc) } @Override + public boolean existPartitions(String databaseName, String tableName) throws CatalogException { + Connection conn = null; + ResultSet res = null; + PreparedStatement pstmt = null; + boolean result = false; + + try { + String sql = "SELECT COUNT CNT FROM " + + TB_PARTTIONS +" WHERE " + COL_TABLES_PK + " = ? "; + + if (LOG.isDebugEnabled()) { + LOG.debug(sql); + } + + int databaseId = getDatabaseId(databaseName); + int tableId = getTableId(databaseId, databaseName, tableName); + + conn = getConnection(); + pstmt = conn.prepareStatement(sql); + pstmt.setInt(1, tableId); + res = pstmt.executeQuery(); + + if (res.next()) { + if (res.getInt("CNT") > 0) { + result = true; + } + } + } catch (SQLException se) { + throw new TajoInternalError(se); + } finally { + CatalogUtil.closeQuietly(pstmt, res); + } + return result; + } + + @Override + public List<TablePartitionProto> getPartitionsByDirectSql(GetPartitionsWithDirectSQLRequest request) — End diff – Catalog API is public. In terms of this point, this API design does not make sense due to the following problems: Users should know table schemas, their relationship, which underlying persistent storage. Users should make SQL statements for all persistent storages. This approach is very prone to SQL injection. The solution is that you can use some intermediate representation, using string or java objects. They should be transformed into SQL statements depending on persistent storages in each catalog driver.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user blrunner commented on the pull request:

          https://github.com/apache/tajo/pull/624#issuecomment-134106861

          Hi @hyunsik

          I've implemented partition API using algebra expressions. For this feature, I implemented some visitors as following:

          • ScanQualConverter: This converts Quals of ScanNode to algebra expressions.
          • PartitionFilterAlgebraVisitor: This build SQL statement for getting partition filters by visiting algebra expressions.
          • PartitionFilterAlgebraVisitor: This build SQL statement for getting partition filters by visiting EvalNodes.
          Show
          githubbot ASF GitHub Bot added a comment - Github user blrunner commented on the pull request: https://github.com/apache/tajo/pull/624#issuecomment-134106861 Hi @hyunsik I've implemented partition API using algebra expressions. For this feature, I implemented some visitors as following: ScanQualConverter: This converts Quals of ScanNode to algebra expressions. PartitionFilterAlgebraVisitor: This build SQL statement for getting partition filters by visiting algebra expressions. PartitionFilterAlgebraVisitor: This build SQL statement for getting partition filters by visiting EvalNodes.
          Hide
          tajoqa Tajo QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12753733/TAJO-1493.patch
          against master revision release-0.9.0-rc0-436-gace0480.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

          +1 checkstyle. The patch generated 0 code style errors.

          -1 findbugs. The patch appears to cause Findbugs (version 2.0.3) to fail.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in tajo-catalog/tajo-catalog-client tajo-catalog/tajo-catalog-common tajo-catalog/tajo-catalog-drivers/tajo-hive tajo-catalog/tajo-catalog-server tajo-common tajo-core tajo-core-tests tajo-plan.

          Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/840//testReport/
          Findbugs results: https://builds.apache.org/job/PreCommit-TAJO-Build/840//findbugsResult
          Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/840//console

          This message is automatically generated.

          Show
          tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12753733/TAJO-1493.patch against master revision release-0.9.0-rc0-436-gace0480. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. -1 findbugs. The patch appears to cause Findbugs (version 2.0.3) to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in tajo-catalog/tajo-catalog-client tajo-catalog/tajo-catalog-common tajo-catalog/tajo-catalog-drivers/tajo-hive tajo-catalog/tajo-catalog-server tajo-common tajo-core tajo-core-tests tajo-plan. Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/840//testReport/ Findbugs results: https://builds.apache.org/job/PreCommit-TAJO-Build/840//findbugsResult Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/840//console This message is automatically generated.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user blrunner commented on the pull request:

          https://github.com/apache/tajo/pull/624#issuecomment-137321819

          I added DateTime types and more unit test cases for various partition column types.

          Show
          githubbot ASF GitHub Bot added a comment - Github user blrunner commented on the pull request: https://github.com/apache/tajo/pull/624#issuecomment-137321819 I added DateTime types and more unit test cases for various partition column types.
          Hide
          blrunner Jaehwa Jung added a comment -

          Added DateTime types and more unit test cases for various partition column types.

          Show
          blrunner Jaehwa Jung added a comment - Added DateTime types and more unit test cases for various partition column types.
          Hide
          tajoqa Tajo QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12753913/TAJO-1493_2.patch
          against master revision release-0.9.0-rc0-437-g22ab1cf.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

          +1 checkstyle. The patch generated 0 code style errors.

          -1 findbugs. The patch appears to cause Findbugs (version 2.0.3) to fail.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in tajo-catalog/tajo-catalog-client tajo-catalog/tajo-catalog-common tajo-catalog/tajo-catalog-drivers/tajo-hive tajo-catalog/tajo-catalog-server tajo-common tajo-core tajo-core-tests tajo-plan.

          Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/842//testReport/
          Findbugs results: https://builds.apache.org/job/PreCommit-TAJO-Build/842//findbugsResult
          Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/842//console

          This message is automatically generated.

          Show
          tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12753913/TAJO-1493_2.patch against master revision release-0.9.0-rc0-437-g22ab1cf. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. -1 findbugs. The patch appears to cause Findbugs (version 2.0.3) to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in tajo-catalog/tajo-catalog-client tajo-catalog/tajo-catalog-common tajo-catalog/tajo-catalog-drivers/tajo-hive tajo-catalog/tajo-catalog-server tajo-common tajo-core tajo-core-tests tajo-plan. Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/842//testReport/ Findbugs results: https://builds.apache.org/job/PreCommit-TAJO-Build/842//findbugsResult Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/842//console This message is automatically generated.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user blrunner commented on the pull request:

          https://github.com/apache/tajo/pull/624#issuecomment-137522218

          I've finished tests successfully for this PR as following:

          • Data: TPC-H benchmark set (scale factor = 1)
          • CatalogStore: MySQLStore, HiveCatalogStore(with apache hive 1.1.0)
          • CTAS for creating partition tables (except partsupp)
            ```
            CREATE TABLE LINEITEM_P (
            L_ORDERKEY bigint, L_PARTKEY bigint, L_SUPPKEY bigint, L_LINENUMBER bigint, L_QUANTITY double, L_EXTENDEDPRICE double, L_DISCOUNT double, L_TAX double, L_RETURNFLAG text, L_LINESTATUS text,L_COMMITDATE date, L_RECEIPTDATE date, L_SHIPINSTRUCT text, L_SHIPMODE text, L_COMMENT text)
            USING TEXT WITH ('text.delimiter'='|')
            PARTITION BY COLUMN (L_SHIPDATE date) AS
            SELECT L_ORDERKEY, L_PARTKEY, L_SUPPKEY, L_LINENUMBER, L_QUANTITY, L_EXTENDEDPRICE, L_DISCOUNT, L_TAX, L_RETURNFLAG, L_LINESTATUS, L_COMMITDATE, L_RECEIPTDATE, L_SHIPINSTRUCT, L_SHIPMODE, L_COMMENT, L_SHIPDATE FROM LINEITEM;

          create table customer_p (C_CUSTKEY bigint, C_NAME text, C_ADDRESS text, C_PHONE text, C_ACCTBAL double, C_MKTSEGMENT text, C_COMMENT text)
          USING TEXT WITH ('text.delimiter'='|')
          PARTITION BY COLUMN (C_NATIONKEY bigint) AS
          SELECT C_CUSTKEY, C_NAME, C_ADDRESS, C_PHONE, C_ACCTBAL, C_MKTSEGMENT, C_COMMENT, C_NATIONKEY FROM customer;

          create  table supplier_p (S_SUPPKEY bigint, S_NAME text, S_ADDRESS text, S_PHONE text, S_ACCTBAL double, S_COMMENT text)
          USING TEXT WITH ('text.delimiter'='|')
          PARTITION BY COLUMN ( S_NATIONKEY bigint) AS
          SELECT S_SUPPKEY, S_NAME, S_ADDRESS, S_PHONE, S_ACCTBAL, S_COMMENT, S_NATIONKEY FROM supplier;

          create  table part_p (P_PARTKEY bigint, P_NAME text, P_MFGR text, P_BRAND text, P_TYPE text, P_CONTAINER text, P_RETAILPRICE double, P_COMMENT text)
          USING TEXT WITH ('text.delimiter'='|')
          PARTITION BY COLUMN (P_SIZE integer) AS
          SELECT P_PARTKEY, P_NAME, P_MFGR, P_BRAND, P_TYPE, P_CONTAINER, P_RETAILPRICE, P_COMMENT, P_SIZE FROM part;

          create table orders_p (O_ORDERKEY bigint, O_CUSTKEY bigint, O_ORDERSTATUS text, O_TOTALPRICE double, O_ORDERPRIORITY text, O_CLERK text, O_SHIPPRIORITY int, O_COMMENT text)
          USING TEXT WITH ('text.delimiter'='|')
          PARTITION BY COLUMN (O_ORDERDATE date) AS
          SELECT O_ORDERKEY, O_CUSTKEY, O_ORDERSTATUS, O_TOTALPRICE, O_ORDERPRIORITY, O_CLERK, O_SHIPPRIORITY, O_COMMENT, O_ORDERDATE FROM orders;

          create table nation_p (N_NATIONKEY bigint, N_NAME text, N_COMMENT text)
          USING TEXT WITH ('text.delimiter'='|')
          PARTITION BY COLUMN (N_REGIONKEY bigint) AS
          SELECT N_NATIONKEY, N_NAME, N_COMMENT, N_REGIONKEY FROM nation;

          create table region_p ( R_NAME text,  R_COMMENT text)
          USING TEXT WITH ('text.delimiter'='|')
          PARTITION BY COLUMN (R_REGIONKEY bigint) AS
          SELECT R_NAME,  R_COMMENT, R_REGIONKEY FROM region;
          ```

          • Test Queries : TPC-H Q1 ~ Q22

          After executing CTAS statements, I fount that list of table partitions had been printed on hive shell. All test results mostly accorded with results of queries for non-partitoned tables in MySQLStore and HiveCatalogStore. But some records for double type record aggregation were slightly different in the nearest whole number, To resolve above problem, we need to provide decimal column type.

          Show
          githubbot ASF GitHub Bot added a comment - Github user blrunner commented on the pull request: https://github.com/apache/tajo/pull/624#issuecomment-137522218 I've finished tests successfully for this PR as following: Data: TPC-H benchmark set (scale factor = 1) CatalogStore: MySQLStore, HiveCatalogStore(with apache hive 1.1.0) CTAS for creating partition tables (except partsupp) ``` CREATE TABLE LINEITEM_P ( L_ORDERKEY bigint, L_PARTKEY bigint, L_SUPPKEY bigint, L_LINENUMBER bigint, L_QUANTITY double, L_EXTENDEDPRICE double, L_DISCOUNT double, L_TAX double, L_RETURNFLAG text, L_LINESTATUS text,L_COMMITDATE date, L_RECEIPTDATE date, L_SHIPINSTRUCT text, L_SHIPMODE text, L_COMMENT text) USING TEXT WITH ('text.delimiter'='|') PARTITION BY COLUMN (L_SHIPDATE date) AS SELECT L_ORDERKEY, L_PARTKEY, L_SUPPKEY, L_LINENUMBER, L_QUANTITY, L_EXTENDEDPRICE, L_DISCOUNT, L_TAX, L_RETURNFLAG, L_LINESTATUS, L_COMMITDATE, L_RECEIPTDATE, L_SHIPINSTRUCT, L_SHIPMODE, L_COMMENT, L_SHIPDATE FROM LINEITEM; create table customer_p (C_CUSTKEY bigint, C_NAME text, C_ADDRESS text, C_PHONE text, C_ACCTBAL double, C_MKTSEGMENT text, C_COMMENT text) USING TEXT WITH ('text.delimiter'='|') PARTITION BY COLUMN (C_NATIONKEY bigint) AS SELECT C_CUSTKEY, C_NAME, C_ADDRESS, C_PHONE, C_ACCTBAL, C_MKTSEGMENT, C_COMMENT, C_NATIONKEY FROM customer; create  table supplier_p (S_SUPPKEY bigint, S_NAME text, S_ADDRESS text, S_PHONE text, S_ACCTBAL double, S_COMMENT text) USING TEXT WITH ('text.delimiter'='|') PARTITION BY COLUMN ( S_NATIONKEY bigint) AS SELECT S_SUPPKEY, S_NAME, S_ADDRESS, S_PHONE, S_ACCTBAL, S_COMMENT, S_NATIONKEY FROM supplier; create  table part_p (P_PARTKEY bigint, P_NAME text, P_MFGR text, P_BRAND text, P_TYPE text, P_CONTAINER text, P_RETAILPRICE double, P_COMMENT text) USING TEXT WITH ('text.delimiter'='|') PARTITION BY COLUMN (P_SIZE integer) AS SELECT P_PARTKEY, P_NAME, P_MFGR, P_BRAND, P_TYPE, P_CONTAINER, P_RETAILPRICE, P_COMMENT, P_SIZE FROM part; create table orders_p (O_ORDERKEY bigint, O_CUSTKEY bigint, O_ORDERSTATUS text, O_TOTALPRICE double, O_ORDERPRIORITY text, O_CLERK text, O_SHIPPRIORITY int, O_COMMENT text) USING TEXT WITH ('text.delimiter'='|') PARTITION BY COLUMN (O_ORDERDATE date) AS SELECT O_ORDERKEY, O_CUSTKEY, O_ORDERSTATUS, O_TOTALPRICE, O_ORDERPRIORITY, O_CLERK, O_SHIPPRIORITY, O_COMMENT, O_ORDERDATE FROM orders; create table nation_p (N_NATIONKEY bigint, N_NAME text, N_COMMENT text) USING TEXT WITH ('text.delimiter'='|') PARTITION BY COLUMN (N_REGIONKEY bigint) AS SELECT N_NATIONKEY, N_NAME, N_COMMENT, N_REGIONKEY FROM nation; create table region_p ( R_NAME text,  R_COMMENT text) USING TEXT WITH ('text.delimiter'='|') PARTITION BY COLUMN (R_REGIONKEY bigint) AS SELECT R_NAME,  R_COMMENT, R_REGIONKEY FROM region; ``` Test Queries : TPC-H Q1 ~ Q22 After executing CTAS statements, I fount that list of table partitions had been printed on hive shell. All test results mostly accorded with results of queries for non-partitoned tables in MySQLStore and HiveCatalogStore. But some records for double type record aggregation were slightly different in the nearest whole number, To resolve above problem, we need to provide decimal column type.
          Hide
          tajoqa Tajo QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12754037/TAJO-1493_3.patch
          against master revision release-0.9.0-rc0-442-gd794c1d.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

          +1 checkstyle. The patch generated 0 code style errors.

          -1 findbugs. The patch appears to cause Findbugs (version 2.0.3) to fail.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in tajo-catalog/tajo-catalog-client tajo-catalog/tajo-catalog-common tajo-catalog/tajo-catalog-drivers/tajo-hive tajo-catalog/tajo-catalog-server tajo-common tajo-core tajo-core-tests tajo-plan.

          Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/847//testReport/
          Findbugs results: https://builds.apache.org/job/PreCommit-TAJO-Build/847//findbugsResult
          Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/847//console

          This message is automatically generated.

          Show
          tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12754037/TAJO-1493_3.patch against master revision release-0.9.0-rc0-442-gd794c1d. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. -1 findbugs. The patch appears to cause Findbugs (version 2.0.3) to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in tajo-catalog/tajo-catalog-client tajo-catalog/tajo-catalog-common tajo-catalog/tajo-catalog-drivers/tajo-hive tajo-catalog/tajo-catalog-server tajo-common tajo-core tajo-core-tests tajo-plan. Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/847//testReport/ Findbugs results: https://builds.apache.org/job/PreCommit-TAJO-Build/847//findbugsResult Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/847//console This message is automatically generated.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/624#discussion_r38730288

          — Diff: tajo-catalog/tajo-catalog-client/src/main/java/org/apache/tajo/catalog/AbstractCatalogClient.java —
          @@ -428,13 +429,12 @@ public final PartitionDescProto getPartition(final String databaseName, final St
          }

          @Override

          • public final List<PartitionDescProto> getPartitions(final String databaseName, final String tableName) {
            + public final List<PartitionDescProto> getPartitions(final String databaseName, final String tableName) throws
            + UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException,
              • End diff –

          To handle such exceptions, you should put ``throwsIfThisError`` method for the exceptions.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/624#discussion_r38730288 — Diff: tajo-catalog/tajo-catalog-client/src/main/java/org/apache/tajo/catalog/AbstractCatalogClient.java — @@ -428,13 +429,12 @@ public final PartitionDescProto getPartition(final String databaseName, final St } @Override public final List<PartitionDescProto> getPartitions(final String databaseName, final String tableName) { + public final List<PartitionDescProto> getPartitions(final String databaseName, final String tableName) throws + UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException, End diff – To handle such exceptions, you should put ``throwsIfThisError`` method for the exceptions.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/624#discussion_r38730328

          — Diff: tajo-catalog/tajo-catalog-client/src/main/java/org/apache/tajo/catalog/AbstractCatalogClient.java —
          @@ -446,6 +446,46 @@ public final PartitionDescProto getPartition(final String databaseName, final St
          }

          @Override
          + public boolean existPartitions(String databaseName, String tableName) {
          + try

          { + final BlockingInterface stub = getStub(); + final TableIdentifierProto request = buildTableIdentifier(databaseName, tableName); + return isSuccess(stub.existPartitionsByTableName(null, request)); + }

          catch (ServiceException e)

          { + throw new RuntimeException(e); + }

          + }
          +
          + @Override
          + public List<PartitionDescProto> getPartitionsByAlgebra(PartitionsByAlgebraProto request) throws
          + UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException,
          — End diff –

          To handle such exceptions, you should put throwsIfThisError method for the exceptions. Please refer to https://github.com/apache/tajo/pull/624/files#diff-9ecb2222e062d0cc008fb1c3da61d4b0L483.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/624#discussion_r38730328 — Diff: tajo-catalog/tajo-catalog-client/src/main/java/org/apache/tajo/catalog/AbstractCatalogClient.java — @@ -446,6 +446,46 @@ public final PartitionDescProto getPartition(final String databaseName, final St } @Override + public boolean existPartitions(String databaseName, String tableName) { + try { + final BlockingInterface stub = getStub(); + final TableIdentifierProto request = buildTableIdentifier(databaseName, tableName); + return isSuccess(stub.existPartitionsByTableName(null, request)); + } catch (ServiceException e) { + throw new RuntimeException(e); + } + } + + @Override + public List<PartitionDescProto> getPartitionsByAlgebra(PartitionsByAlgebraProto request) throws + UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException, — End diff – To handle such exceptions, you should put throwsIfThisError method for the exceptions. Please refer to https://github.com/apache/tajo/pull/624/files#diff-9ecb2222e062d0cc008fb1c3da61d4b0L483 .
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/624#discussion_r38730338

          — Diff: tajo-catalog/tajo-catalog-client/src/main/java/org/apache/tajo/catalog/AbstractCatalogClient.java —
          @@ -446,6 +446,46 @@ public final PartitionDescProto getPartition(final String databaseName, final St
          }

          @Override
          + public boolean existPartitions(String databaseName, String tableName) {
          + try

          { + final BlockingInterface stub = getStub(); + final TableIdentifierProto request = buildTableIdentifier(databaseName, tableName); + return isSuccess(stub.existPartitionsByTableName(null, request)); + }

          catch (ServiceException e)

          { + throw new RuntimeException(e); + }

          + }
          +
          + @Override
          + public List<PartitionDescProto> getPartitionsByAlgebra(PartitionsByAlgebraProto request) throws
          + UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException,
          + UndefinedOperatorException {
          + try

          { + final BlockingInterface stub = getStub(); + GetPartitionsResponse response = stub.getPartitionsByAlgebra(null, request); + ensureOk(response.getState()); + return response.getPartitionList(); + }

          catch (ServiceException e)

          { + LOG.error(e.getMessage(), e); + return null; + }

          + }
          +
          + @Override
          + public List<PartitionDescProto> getPartitionsByDirectSql(PartitionsByDirectSqlProto request) throws
          + UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException,
          — End diff –

          To handle such exceptions, you should put throwsIfThisError method for the exceptions. Please refer to https://github.com/apache/tajo/pull/624/files#diff-9ecb2222e062d0cc008fb1c3da61d4b0L483.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/624#discussion_r38730338 — Diff: tajo-catalog/tajo-catalog-client/src/main/java/org/apache/tajo/catalog/AbstractCatalogClient.java — @@ -446,6 +446,46 @@ public final PartitionDescProto getPartition(final String databaseName, final St } @Override + public boolean existPartitions(String databaseName, String tableName) { + try { + final BlockingInterface stub = getStub(); + final TableIdentifierProto request = buildTableIdentifier(databaseName, tableName); + return isSuccess(stub.existPartitionsByTableName(null, request)); + } catch (ServiceException e) { + throw new RuntimeException(e); + } + } + + @Override + public List<PartitionDescProto> getPartitionsByAlgebra(PartitionsByAlgebraProto request) throws + UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException, + UndefinedOperatorException { + try { + final BlockingInterface stub = getStub(); + GetPartitionsResponse response = stub.getPartitionsByAlgebra(null, request); + ensureOk(response.getState()); + return response.getPartitionList(); + } catch (ServiceException e) { + LOG.error(e.getMessage(), e); + return null; + } + } + + @Override + public List<PartitionDescProto> getPartitionsByDirectSql(PartitionsByDirectSqlProto request) throws + UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException, — End diff – To handle such exceptions, you should put throwsIfThisError method for the exceptions. Please refer to https://github.com/apache/tajo/pull/624/files#diff-9ecb2222e062d0cc008fb1c3da61d4b0L483 .
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/624#discussion_r38730473

          — Diff: tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogService.java —
          @@ -162,7 +162,19 @@ PartitionDescProto getPartition(String databaseName, String tableName, String pa
          throws UndefinedPartitionException, UndefinedPartitionMethodException, UndefinedDatabaseException,
          UndefinedTableException;

          • List<PartitionDescProto> getPartitions(String databaseName, String tableName);
            + List<PartitionDescProto> getPartitions(String databaseName, String tableName) throws UndefinedDatabaseException,
              • End diff –

          It would be better if ``getPartitions()`` is renamed to ``getAllPartitions()``.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/624#discussion_r38730473 — Diff: tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogService.java — @@ -162,7 +162,19 @@ PartitionDescProto getPartition(String databaseName, String tableName, String pa throws UndefinedPartitionException, UndefinedPartitionMethodException, UndefinedDatabaseException, UndefinedTableException; List<PartitionDescProto> getPartitions(String databaseName, String tableName); + List<PartitionDescProto> getPartitions(String databaseName, String tableName) throws UndefinedDatabaseException, End diff – It would be better if ``getPartitions()`` is renamed to ``getAllPartitions()``.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/624#discussion_r38730495

          — Diff: tajo-catalog/tajo-catalog-client/src/main/java/org/apache/tajo/catalog/AbstractCatalogClient.java —
          @@ -428,13 +429,12 @@ public final PartitionDescProto getPartition(final String databaseName, final St
          }

          @Override

          • public final List<PartitionDescProto> getPartitions(final String databaseName, final String tableName) {
            + public final List<PartitionDescProto> getPartitions(final String databaseName, final String tableName) throws
              • End diff –

          It would be better if ``getPartitions()`` is renamed to ``getAllPartitions()``.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/624#discussion_r38730495 — Diff: tajo-catalog/tajo-catalog-client/src/main/java/org/apache/tajo/catalog/AbstractCatalogClient.java — @@ -428,13 +429,12 @@ public final PartitionDescProto getPartition(final String databaseName, final St } @Override public final List<PartitionDescProto> getPartitions(final String databaseName, final String tableName) { + public final List<PartitionDescProto> getPartitions(final String databaseName, final String tableName) throws End diff – It would be better if ``getPartitions()`` is renamed to ``getAllPartitions()``.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/624#discussion_r38730634

          — Diff: tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogStore.java —
          @@ -846,10 +855,65 @@ public boolean existPartitionMethod(String databaseName, String tableName) throw

          @Override
          public List<CatalogProtos.PartitionDescProto> getPartitions(String databaseName,

          • String tableName) {
            + String tableName) throws UndefinedDatabaseException,
            + UndefinedTableException, UndefinedPartitionMethodException, UndefinedPartitionException { throw new UnsupportedOperationException(); }

          + @Override
          + public boolean existPartitions(String databaseName, String tableName) throws UndefinedDatabaseException,
          + UndefinedTableException, UndefinedPartitionMethodException

          { + throw new UnsupportedOperationException(); + }

          +
          + @Override
          + public List<PartitionDescProto> getPartitionsByAlgebra(PartitionsByAlgebraProto request)
          + throws UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException,
          + UndefinedOperatorException {
          + throw new UndefinedOperatorException("getPartitionsByAlgebra");
          — End diff –

          UnsupportedException is proper here.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/624#discussion_r38730634 — Diff: tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogStore.java — @@ -846,10 +855,65 @@ public boolean existPartitionMethod(String databaseName, String tableName) throw @Override public List<CatalogProtos.PartitionDescProto> getPartitions(String databaseName, String tableName) { + String tableName) throws UndefinedDatabaseException, + UndefinedTableException, UndefinedPartitionMethodException, UndefinedPartitionException { throw new UnsupportedOperationException(); } + @Override + public boolean existPartitions(String databaseName, String tableName) throws UndefinedDatabaseException, + UndefinedTableException, UndefinedPartitionMethodException { + throw new UnsupportedOperationException(); + } + + @Override + public List<PartitionDescProto> getPartitionsByAlgebra(PartitionsByAlgebraProto request) + throws UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException, + UndefinedOperatorException { + throw new UndefinedOperatorException("getPartitionsByAlgebra"); — End diff – UnsupportedException is proper here.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/624#discussion_r38730640

          — Diff: tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogStore.java —
          @@ -846,10 +855,65 @@ public boolean existPartitionMethod(String databaseName, String tableName) throw

          @Override
          public List<CatalogProtos.PartitionDescProto> getPartitions(String databaseName,

          • String tableName) {
            + String tableName) throws UndefinedDatabaseException,
            + UndefinedTableException, UndefinedPartitionMethodException, UndefinedPartitionException { throw new UnsupportedOperationException(); }

          + @Override
          + public boolean existPartitions(String databaseName, String tableName) throws UndefinedDatabaseException,
          + UndefinedTableException, UndefinedPartitionMethodException {
          + throw new UnsupportedOperationException();
          — End diff –

          UnsupportedException is proper here.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/624#discussion_r38730640 — Diff: tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogStore.java — @@ -846,10 +855,65 @@ public boolean existPartitionMethod(String databaseName, String tableName) throw @Override public List<CatalogProtos.PartitionDescProto> getPartitions(String databaseName, String tableName) { + String tableName) throws UndefinedDatabaseException, + UndefinedTableException, UndefinedPartitionMethodException, UndefinedPartitionException { throw new UnsupportedOperationException(); } + @Override + public boolean existPartitions(String databaseName, String tableName) throws UndefinedDatabaseException, + UndefinedTableException, UndefinedPartitionMethodException { + throw new UnsupportedOperationException(); — End diff – UnsupportedException is proper here.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/624#discussion_r38730647

          — Diff: tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogStore.java —
          @@ -846,10 +855,65 @@ public boolean existPartitionMethod(String databaseName, String tableName) throw

          @Override
          public List<CatalogProtos.PartitionDescProto> getPartitions(String databaseName,

          • String tableName) {
            + String tableName) throws UndefinedDatabaseException,
            + UndefinedTableException, UndefinedPartitionMethodException, UndefinedPartitionException{
            throw new UnsupportedOperationException();
              • End diff –

          UnsupportedException is proper here.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/624#discussion_r38730647 — Diff: tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogStore.java — @@ -846,10 +855,65 @@ public boolean existPartitionMethod(String databaseName, String tableName) throw @Override public List<CatalogProtos.PartitionDescProto> getPartitions(String databaseName, String tableName) { + String tableName) throws UndefinedDatabaseException, + UndefinedTableException, UndefinedPartitionMethodException, UndefinedPartitionException{ throw new UnsupportedOperationException(); End diff – UnsupportedException is proper here.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/624#discussion_r38730778

          — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java —
          @@ -2173,6 +2179,244 @@ private void setPartitionKeys(int pid, PartitionDescProto.Builder partitionDesc)
          }

          @Override
          + public boolean existPartitions(String databaseName, String tableName) throws UndefinedDatabaseException,
          + UndefinedTableException, UndefinedPartitionMethodException {
          + Connection conn = null;
          + ResultSet res = null;
          + PreparedStatement pstmt = null;
          + boolean result = false;
          +
          + try {
          + String sql = "SELECT COUNT CNT FROM "
          + + TB_PARTTIONS +" WHERE " + COL_TABLES_PK + " = ? ";
          +
          + if (LOG.isDebugEnabled())

          { + LOG.debug(sql); + }

          +
          + int databaseId = getDatabaseId(databaseName);
          + int tableId = getTableId(databaseId, databaseName, tableName);
          +
          + conn = getConnection();
          + pstmt = conn.prepareStatement(sql);
          + pstmt.setInt(1, tableId);
          + res = pstmt.executeQuery();
          +
          + if (res.next()) {
          + if (res.getInt("CNT") > 0)

          { + result = true; + }

          + }
          + } catch (SQLException se)

          { + throw new TajoInternalError(se); + }

          finally

          { + CatalogUtil.closeQuietly(pstmt, res); + }

          + return result;
          + }
          +
          + @Override
          + public List<PartitionDescProto> getPartitionsByDirectSql(PartitionsByDirectSqlProto request)
          + throws UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException,
          + UndefinedOperatorException {
          + throw new UndefinedOperatorException("getPartitionsByDirectSql");
          — End diff –

          It should be UnsupportedException.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/624#discussion_r38730778 — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java — @@ -2173,6 +2179,244 @@ private void setPartitionKeys(int pid, PartitionDescProto.Builder partitionDesc) } @Override + public boolean existPartitions(String databaseName, String tableName) throws UndefinedDatabaseException, + UndefinedTableException, UndefinedPartitionMethodException { + Connection conn = null; + ResultSet res = null; + PreparedStatement pstmt = null; + boolean result = false; + + try { + String sql = "SELECT COUNT CNT FROM " + + TB_PARTTIONS +" WHERE " + COL_TABLES_PK + " = ? "; + + if (LOG.isDebugEnabled()) { + LOG.debug(sql); + } + + int databaseId = getDatabaseId(databaseName); + int tableId = getTableId(databaseId, databaseName, tableName); + + conn = getConnection(); + pstmt = conn.prepareStatement(sql); + pstmt.setInt(1, tableId); + res = pstmt.executeQuery(); + + if (res.next()) { + if (res.getInt("CNT") > 0) { + result = true; + } + } + } catch (SQLException se) { + throw new TajoInternalError(se); + } finally { + CatalogUtil.closeQuietly(pstmt, res); + } + return result; + } + + @Override + public List<PartitionDescProto> getPartitionsByDirectSql(PartitionsByDirectSqlProto request) + throws UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException, + UndefinedOperatorException { + throw new UndefinedOperatorException("getPartitionsByDirectSql"); — End diff – It should be UnsupportedException.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/624#discussion_r38730836

          — Diff: tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogStore.java —
          @@ -846,10 +855,65 @@ public boolean existPartitionMethod(String databaseName, String tableName) throw

          @Override
          public List<CatalogProtos.PartitionDescProto> getPartitions(String databaseName,

          • String tableName) {
            + String tableName) throws UndefinedDatabaseException,
            + UndefinedTableException, UndefinedPartitionMethodException, UndefinedPartitionException { throw new UnsupportedOperationException(); }

          + @Override
          + public boolean existPartitions(String databaseName, String tableName) throws UndefinedDatabaseException,
          + UndefinedTableException, UndefinedPartitionMethodException

          { + throw new UnsupportedOperationException(); + }

          +
          + @Override
          + public List<PartitionDescProto> getPartitionsByAlgebra(PartitionsByAlgebraProto request)
          + throws UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException,
          + UndefinedOperatorException

          { + throw new UndefinedOperatorException("getPartitionsByAlgebra"); + }

          +
          + @Override
          + public List<PartitionDescProto> getPartitionsByDirectSql(PartitionsByDirectSqlProto request)
          + throws UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException,
          + UndefinedOperatorException {
          — End diff –

          UndefinedOperator is not proper here.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/624#discussion_r38730836 — Diff: tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogStore.java — @@ -846,10 +855,65 @@ public boolean existPartitionMethod(String databaseName, String tableName) throw @Override public List<CatalogProtos.PartitionDescProto> getPartitions(String databaseName, String tableName) { + String tableName) throws UndefinedDatabaseException, + UndefinedTableException, UndefinedPartitionMethodException, UndefinedPartitionException { throw new UnsupportedOperationException(); } + @Override + public boolean existPartitions(String databaseName, String tableName) throws UndefinedDatabaseException, + UndefinedTableException, UndefinedPartitionMethodException { + throw new UnsupportedOperationException(); + } + + @Override + public List<PartitionDescProto> getPartitionsByAlgebra(PartitionsByAlgebraProto request) + throws UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException, + UndefinedOperatorException { + throw new UndefinedOperatorException("getPartitionsByAlgebra"); + } + + @Override + public List<PartitionDescProto> getPartitionsByDirectSql(PartitionsByDirectSqlProto request) + throws UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException, + UndefinedOperatorException { — End diff – UndefinedOperator is not proper here.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/624#discussion_r38730855

          — Diff: tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogStore.java —
          @@ -846,10 +855,65 @@ public boolean existPartitionMethod(String databaseName, String tableName) throw

          @Override
          public List<CatalogProtos.PartitionDescProto> getPartitions(String databaseName,

          • String tableName) {
            + String tableName) throws UndefinedDatabaseException,
            + UndefinedTableException, UndefinedPartitionMethodException, UndefinedPartitionException { throw new UnsupportedOperationException(); }

          + @Override
          + public boolean existPartitions(String databaseName, String tableName) throws UndefinedDatabaseException,
          + UndefinedTableException, UndefinedPartitionMethodException

          { + throw new UnsupportedOperationException(); + }

          +
          + @Override
          + public List<PartitionDescProto> getPartitionsByAlgebra(PartitionsByAlgebraProto request)
          + throws UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException,
          + UndefinedOperatorException

          { + throw new UndefinedOperatorException("getPartitionsByAlgebra"); + }

          +
          + @Override
          + public List<PartitionDescProto> getPartitionsByDirectSql(PartitionsByDirectSqlProto request)
          + throws UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException,
          + UndefinedOperatorException {
          +
          + HiveCatalogStoreClientPool.HiveCatalogStoreClient client = null;
          + List<PartitionDescProto> partitions = null;
          +
          + try {
          + String databaseName = request.getDatabaseName();
          + String tableName = request.getTableName();
          +
          + partitions = TUtil.newList();
          +
          + client = clientPool.getClient();
          +
          + List<Partition> hivePartitions = client.getHiveClient().listPartitionsByFilter(databaseName, tableName
          + , request.getDirectSql(), (short) -1);
          +
          + for (Partition hivePartition : hivePartitions)

          { + CatalogProtos.PartitionDescProto.Builder builder = CatalogProtos.PartitionDescProto.newBuilder(); + builder.setPath(hivePartition.getSd().getLocation()); + + int startIndex = hivePartition.getSd().getLocation().indexOf(tableName) + tableName.length(); + String partitionName = hivePartition.getSd().getLocation().substring(startIndex+1); + builder.setPartitionName(partitionName); + + partitions.add(builder.build()); + }

          + } catch (NoSuchObjectException e) {
          + return null;
          — End diff –

          This should throw TajoInternalError.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/624#discussion_r38730855 — Diff: tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogStore.java — @@ -846,10 +855,65 @@ public boolean existPartitionMethod(String databaseName, String tableName) throw @Override public List<CatalogProtos.PartitionDescProto> getPartitions(String databaseName, String tableName) { + String tableName) throws UndefinedDatabaseException, + UndefinedTableException, UndefinedPartitionMethodException, UndefinedPartitionException { throw new UnsupportedOperationException(); } + @Override + public boolean existPartitions(String databaseName, String tableName) throws UndefinedDatabaseException, + UndefinedTableException, UndefinedPartitionMethodException { + throw new UnsupportedOperationException(); + } + + @Override + public List<PartitionDescProto> getPartitionsByAlgebra(PartitionsByAlgebraProto request) + throws UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException, + UndefinedOperatorException { + throw new UndefinedOperatorException("getPartitionsByAlgebra"); + } + + @Override + public List<PartitionDescProto> getPartitionsByDirectSql(PartitionsByDirectSqlProto request) + throws UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException, + UndefinedOperatorException { + + HiveCatalogStoreClientPool.HiveCatalogStoreClient client = null; + List<PartitionDescProto> partitions = null; + + try { + String databaseName = request.getDatabaseName(); + String tableName = request.getTableName(); + + partitions = TUtil.newList(); + + client = clientPool.getClient(); + + List<Partition> hivePartitions = client.getHiveClient().listPartitionsByFilter(databaseName, tableName + , request.getDirectSql(), (short) -1); + + for (Partition hivePartition : hivePartitions) { + CatalogProtos.PartitionDescProto.Builder builder = CatalogProtos.PartitionDescProto.newBuilder(); + builder.setPath(hivePartition.getSd().getLocation()); + + int startIndex = hivePartition.getSd().getLocation().indexOf(tableName) + tableName.length(); + String partitionName = hivePartition.getSd().getLocation().substring(startIndex+1); + builder.setPartitionName(partitionName); + + partitions.add(builder.build()); + } + } catch (NoSuchObjectException e) { + return null; — End diff – This should throw TajoInternalError.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/624#discussion_r38731533

          — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java —
          @@ -957,7 +957,61 @@ public GetPartitionDescResponse getPartitionByPartitionName(RpcController contro
          }

          @Override

          • public GetPartitionsResponse getPartitionsByTableName(RpcController controller, PartitionIdentifierProto request)
            + public ReturnState existPartitionsByTableName(RpcController controller, TableIdentifierProto request)
            + throws ServiceException {
            + String dbName = request.getDatabaseName();
            + String tbName = request.getTableName();
            +
            + try
            Unknown macro: { + // linked meta data do not support partition. + // So, the request that wants to get partitions in this db will be failed. + if (linkedMetadataManager.existsDatabase(dbName)) { + return errUndefinedPartitionMethod(tbName); + } + }

            catch (Throwable t)

            { + printStackTraceIfError(LOG, t); + return returnError(t); + }

            +
            + if (metaDictionary.isSystemDatabase(dbName))

            { + return errUndefinedPartitionMethod(tbName); + }

            +
            + rlock.lock();
            + try {
            + boolean contain;
            +
            + contain = store.existDatabase(dbName);
            + if (contain) {

              • End diff –

          These tests can be improved like other methods. Please refer to existsDatabase() method.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/624#discussion_r38731533 — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java — @@ -957,7 +957,61 @@ public GetPartitionDescResponse getPartitionByPartitionName(RpcController contro } @Override public GetPartitionsResponse getPartitionsByTableName(RpcController controller, PartitionIdentifierProto request) + public ReturnState existPartitionsByTableName(RpcController controller, TableIdentifierProto request) + throws ServiceException { + String dbName = request.getDatabaseName(); + String tbName = request.getTableName(); + + try Unknown macro: { + // linked meta data do not support partition. + // So, the request that wants to get partitions in this db will be failed. + if (linkedMetadataManager.existsDatabase(dbName)) { + return errUndefinedPartitionMethod(tbName); + } + } catch (Throwable t) { + printStackTraceIfError(LOG, t); + return returnError(t); + } + + if (metaDictionary.isSystemDatabase(dbName)) { + return errUndefinedPartitionMethod(tbName); + } + + rlock.lock(); + try { + boolean contain; + + contain = store.existDatabase(dbName); + if (contain) { End diff – These tests can be improved like other methods. Please refer to existsDatabase() method.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/624#discussion_r38731577

          — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java —
          @@ -1027,6 +1081,136 @@ public GetTablePartitionsResponse getAllPartitions(RpcController controller, Nul
          }

          @Override
          + public GetPartitionsResponse getPartitionsByAlgebra(RpcController controller,
          + PartitionsByAlgebraProto request) throws ServiceException {
          + String dbName = request.getDatabaseName();
          + String tbName = request.getTableName();
          +
          + try {
          + // linked meta data do not support partition.
          + // So, the request that wants to get partitions in this db will be failed.
          + if (linkedMetadataManager.existsDatabase(dbName))

          { + return GetPartitionsResponse.newBuilder().setState(errUndefinedPartitionMethod(tbName)).build(); + }

          + } catch (Throwable t)

          { + printStackTraceIfError(LOG, t); + return GetPartitionsResponse.newBuilder() + .setState(returnError(t)) + .build(); + }

          +
          + if (metaDictionary.isSystemDatabase(dbName))

          { + return GetPartitionsResponse.newBuilder().setState(errUndefinedPartitionMethod(tbName)).build(); + }

          +
          + rlock.lock();
          + try {
          + boolean contain;
          +
          + contain = store.existDatabase(dbName);
          + if (contain) {
          — End diff –

          These tests can be improved like other methods. Please refer to existsDatabase() method.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/624#discussion_r38731577 — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java — @@ -1027,6 +1081,136 @@ public GetTablePartitionsResponse getAllPartitions(RpcController controller, Nul } @Override + public GetPartitionsResponse getPartitionsByAlgebra(RpcController controller, + PartitionsByAlgebraProto request) throws ServiceException { + String dbName = request.getDatabaseName(); + String tbName = request.getTableName(); + + try { + // linked meta data do not support partition. + // So, the request that wants to get partitions in this db will be failed. + if (linkedMetadataManager.existsDatabase(dbName)) { + return GetPartitionsResponse.newBuilder().setState(errUndefinedPartitionMethod(tbName)).build(); + } + } catch (Throwable t) { + printStackTraceIfError(LOG, t); + return GetPartitionsResponse.newBuilder() + .setState(returnError(t)) + .build(); + } + + if (metaDictionary.isSystemDatabase(dbName)) { + return GetPartitionsResponse.newBuilder().setState(errUndefinedPartitionMethod(tbName)).build(); + } + + rlock.lock(); + try { + boolean contain; + + contain = store.existDatabase(dbName); + if (contain) { — End diff – These tests can be improved like other methods. Please refer to existsDatabase() method.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/624#discussion_r38731587

          — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java —
          @@ -1027,6 +1081,136 @@ public GetTablePartitionsResponse getAllPartitions(RpcController controller, Nul
          }

          @Override
          + public GetPartitionsResponse getPartitionsByAlgebra(RpcController controller,
          + PartitionsByAlgebraProto request) throws ServiceException {
          + String dbName = request.getDatabaseName();
          + String tbName = request.getTableName();
          +
          + try {
          + // linked meta data do not support partition.
          + // So, the request that wants to get partitions in this db will be failed.
          + if (linkedMetadataManager.existsDatabase(dbName))

          { + return GetPartitionsResponse.newBuilder().setState(errUndefinedPartitionMethod(tbName)).build(); + }
          + } catch (Throwable t) { + printStackTraceIfError(LOG, t); + return GetPartitionsResponse.newBuilder() + .setState(returnError(t)) + .build(); + }
          +
          + if (metaDictionary.isSystemDatabase(dbName)) { + return GetPartitionsResponse.newBuilder().setState(errUndefinedPartitionMethod(tbName)).build(); + }
          +
          + rlock.lock();
          + try {
          + boolean contain;
          +
          + contain = store.existDatabase(dbName);
          + if (contain) {
          + contain = store.existTable(dbName, tbName);
          + if (contain) {
          +
          + if (store.existPartitionMethod(dbName, tbName)) { + GetPartitionsResponse.Builder builder = GetPartitionsResponse.newBuilder(); + List<PartitionDescProto> partitions = store.getPartitionsByAlgebra(request); + builder.addAllPartition(partitions); + builder.setState(OK); + return builder.build(); + } else { + return GetPartitionsResponse.newBuilder() + .setState(errUndefinedPartitionMethod(tbName)) + .build(); + }
          + } else { + return GetPartitionsResponse.newBuilder() + .setState(errUndefinedTable(tbName)) + .build(); + }
          + } else { + return GetPartitionsResponse.newBuilder() + .setState(errUndefinedDatabase(dbName)) + .build(); + }
          + } catch (Throwable t) { + printStackTraceIfError(LOG, t); + + return GetPartitionsResponse.newBuilder() + .setState(returnError(t)) + .build(); + + } finally { + rlock.unlock(); + }
          + }
          +
          + @Override
          + public GetPartitionsResponse getPartitionsByDirectSql(RpcController controller,
          + PartitionsByDirectSqlProto request) throws ServiceException {
          + String dbName = request.getDatabaseName();
          + String tbName = request.getTableName();
          +
          + try {
          + // linked meta data do not support partition.
          + // So, the request that wants to get partitions in this db will be failed.
          + if (linkedMetadataManager.existsDatabase(dbName)) { + return GetPartitionsResponse.newBuilder().setState(errUndefinedPartitionMethod(tbName)).build(); + }

          + } catch (Throwable t)

          { + printStackTraceIfError(LOG, t); + return GetPartitionsResponse.newBuilder() + .setState(returnError(t)) + .build(); + }

          +
          + if (metaDictionary.isSystemDatabase(dbName))

          { + return GetPartitionsResponse.newBuilder().setState(errUndefinedPartitionMethod(tbName)).build(); + }

          +
          + rlock.lock();
          + try {
          + boolean contain;
          +
          + contain = store.existDatabase(dbName);
          + if (contain) {
          — End diff –

          These tests can be improved like other methods. Please refer to existsDatabase() method.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/624#discussion_r38731587 — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java — @@ -1027,6 +1081,136 @@ public GetTablePartitionsResponse getAllPartitions(RpcController controller, Nul } @Override + public GetPartitionsResponse getPartitionsByAlgebra(RpcController controller, + PartitionsByAlgebraProto request) throws ServiceException { + String dbName = request.getDatabaseName(); + String tbName = request.getTableName(); + + try { + // linked meta data do not support partition. + // So, the request that wants to get partitions in this db will be failed. + if (linkedMetadataManager.existsDatabase(dbName)) { + return GetPartitionsResponse.newBuilder().setState(errUndefinedPartitionMethod(tbName)).build(); + } + } catch (Throwable t) { + printStackTraceIfError(LOG, t); + return GetPartitionsResponse.newBuilder() + .setState(returnError(t)) + .build(); + } + + if (metaDictionary.isSystemDatabase(dbName)) { + return GetPartitionsResponse.newBuilder().setState(errUndefinedPartitionMethod(tbName)).build(); + } + + rlock.lock(); + try { + boolean contain; + + contain = store.existDatabase(dbName); + if (contain) { + contain = store.existTable(dbName, tbName); + if (contain) { + + if (store.existPartitionMethod(dbName, tbName)) { + GetPartitionsResponse.Builder builder = GetPartitionsResponse.newBuilder(); + List<PartitionDescProto> partitions = store.getPartitionsByAlgebra(request); + builder.addAllPartition(partitions); + builder.setState(OK); + return builder.build(); + } else { + return GetPartitionsResponse.newBuilder() + .setState(errUndefinedPartitionMethod(tbName)) + .build(); + } + } else { + return GetPartitionsResponse.newBuilder() + .setState(errUndefinedTable(tbName)) + .build(); + } + } else { + return GetPartitionsResponse.newBuilder() + .setState(errUndefinedDatabase(dbName)) + .build(); + } + } catch (Throwable t) { + printStackTraceIfError(LOG, t); + + return GetPartitionsResponse.newBuilder() + .setState(returnError(t)) + .build(); + + } finally { + rlock.unlock(); + } + } + + @Override + public GetPartitionsResponse getPartitionsByDirectSql(RpcController controller, + PartitionsByDirectSqlProto request) throws ServiceException { + String dbName = request.getDatabaseName(); + String tbName = request.getTableName(); + + try { + // linked meta data do not support partition. + // So, the request that wants to get partitions in this db will be failed. + if (linkedMetadataManager.existsDatabase(dbName)) { + return GetPartitionsResponse.newBuilder().setState(errUndefinedPartitionMethod(tbName)).build(); + } + } catch (Throwable t) { + printStackTraceIfError(LOG, t); + return GetPartitionsResponse.newBuilder() + .setState(returnError(t)) + .build(); + } + + if (metaDictionary.isSystemDatabase(dbName)) { + return GetPartitionsResponse.newBuilder().setState(errUndefinedPartitionMethod(tbName)).build(); + } + + rlock.lock(); + try { + boolean contain; + + contain = store.existDatabase(dbName); + if (contain) { — End diff – These tests can be improved like other methods. Please refer to existsDatabase() method.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user blrunner commented on the pull request:

          https://github.com/apache/tajo/pull/624#issuecomment-137740521

          @hyunsik

          Thank your for your detailed review.
          I've just updated this PR using your comments.

          Show
          githubbot ASF GitHub Bot added a comment - Github user blrunner commented on the pull request: https://github.com/apache/tajo/pull/624#issuecomment-137740521 @hyunsik Thank your for your detailed review. I've just updated this PR using your comments.
          Hide
          blrunner Jaehwa Jung added a comment -

          Rebased

          Show
          blrunner Jaehwa Jung added a comment - Rebased
          Hide
          tajoqa Tajo QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12754454/TAJO-1493_4.patch
          against master revision release-0.9.0-rc0-447-g33b8893.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 5 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

          +1 checkstyle. The patch generated 0 code style errors.

          -1 findbugs. The patch appears to cause Findbugs (version 2.0.3) to fail.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in tajo-catalog/tajo-catalog-client tajo-catalog/tajo-catalog-common tajo-catalog/tajo-catalog-drivers/tajo-hive tajo-catalog/tajo-catalog-server tajo-common tajo-core tajo-core-tests tajo-plan:
          org.apache.tajo.engine.function.TestBuiltinFunctions
          org.apache.tajo.engine.query.TestCreateTable
          org.apache.tajo.engine.query.TestInnerJoinQuery
          org.apache.tajo.engine.function.TestConditionalExpressions
          org.apache.tajo.engine.query.TestInnerJoinWithSubQuery
          org.apache.tajo.engine.query.TestInsertQuery
          org.apache.tajo.engine.eval.TestSQLExpression
          org.apache.tajo.engine.query.TestNullValues
          org.apache.tajo.engine.query.TestJsonWithTimezone
          org.apache.tajo.engine.query.TestOuterJoinQuery
          org.apache.tajo.engine.query.TestCreateDatabase
          org.apache.tajo.util.history.TestHistoryWriterReader
          org.apache.tajo.engine.function.TestDateTimeFunctions
          org.apache.tajo.engine.query.TestWindowQuery
          org.apache.tajo.engine.query.TestOuterJoinWithSubQuery
          org.apache.tajo.master.TestNonForwardQueryResultSystemScanner
          org.apache.tajo.ws.rs.resources.TestQueryResultResource
          org.apache.tajo.engine.function.TestStringOperatorsAndFunctions
          org.apache.tajo.engine.eval.TestSQLDateTimeTypes
          org.apache.tajo.benchmark.TestTPCH
          org.apache.tajo.engine.planner.TestPlannerUtil
          org.apache.tajo.cli.tsql.commands.TestExecExternalShellCommand
          org.apache.tajo.ws.rs.resources.TestClusterResource
          org.apache.tajo.client.TestQueryClientExceptions
          org.apache.tajo.querymaster.TestQueryState
          org.apache.tajo.ws.rs.resources.TestTablesResource
          org.apache.tajo.engine.eval.TestPredicates
          org.apache.tajo.engine.query.TestAlterTablespace
          org.apache.tajo.ws.rs.resources.TestDatabasesResource
          org.apache.tajo.cli.tsql.TestTajoCli
          org.apache.tajo.engine.query.TestJoinOnPartitionedTables
          org.apache.tajo.engine.query.TestTruncateTable
          org.apache.tajo.engine.query.TestMultipleJoinTypes
          org.apache.tajo.engine.query.TestSelectQuery
          org.apache.tajo.engine.function.TestFunctionLoader
          org.apache.tajo.engine.codegen.TestEvalCodeGenerator
          org.apache.tajo.engine.query.TestInSubquery
          org.apache.tajo.cli.tsql.TestDefaultCliOutputFormatter
          org.apache.tajo.engine.function.TestMathFunctions
          org.apache.tajo.engine.query.TestNetTypes
          org.apache.tajo.engine.query.TestTablePartitions
          org.apache.tajo.engine.eval.TestIntervalType
          org.apache.tajo.master.TestQueryResult
          org.apache.tajo.engine.query.TestIndexScan
          org.apache.tajo.engine.planner.physical.TestSortExec
          org.apache.tajo.engine.query.TestGroupByQuery
          org.apache.tajo.engine.query.TestSetSessionQuery
          org.apache.tajo.engine.query.TestSelectNestedRecord
          org.apache.tajo.engine.function.TestJsonFunctions
          org.apache.tajo.cli.tools.TestTajoDump
          org.apache.tajo.engine.query.TestCreateIndex
          org.apache.tajo.engine.query.TestTableSubQuery
          org.apache.tajo.engine.planner.TestQueryValidation
          org.apache.tajo.engine.query.TestCommonConditionReduce
          org.apache.tajo.engine.query.TestDropTable
          org.apache.tajo.cli.tsql.commands.TestHdfsCommand
          org.apache.tajo.client.v2.TestTajoClientV2
          org.apache.tajo.ws.rs.resources.TestFunctionsResource
          org.apache.tajo.engine.query.TestHBaseTable
          org.apache.tajo.engine.query.TestCrossJoin
          org.apache.tajo.querymaster.TestTaskStatusUpdate
          org.apache.tajo.querymaster.TestKillQuery
          org.apache.tajo.client.TestTajoClient
          org.apache.tajo.client.TestCatalogAdminClientExceptions
          org.apache.tajo.engine.query.TestSimpleQuery
          org.apache.tajo.ws.rs.resources.TestQueryResource
          org.apache.tajo.engine.function.TestPatternMatchingPredicates
          org.apache.tajo.engine.query.TestUnionQuery
          org.apache.tajo.cli.tsql.TestTajoCliNegatives
          org.apache.tajo.engine.function.TestPythonFunctions
          org.apache.tajo.querymaster.TestQueryProgress
          org.apache.tajo.engine.query.TestAlterTable
          org.apache.tajo.engine.query.TestCaseByCases
          org.apache.tajo.ws.rs.resources.TestSessionsResource
          org.apache.tajo.engine.eval.TestEvalTree
          org.apache.tajo.ha.TestHAServiceHDFSImpl
          org.apache.tajo.engine.query.TestSortQuery
          org.apache.tajo.engine.query.TestCTASQuery

          Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/861//testReport/
          Findbugs results: https://builds.apache.org/job/PreCommit-TAJO-Build/861//findbugsResult
          Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/861//console

          This message is automatically generated.

          Show
          tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12754454/TAJO-1493_4.patch against master revision release-0.9.0-rc0-447-g33b8893. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 5 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. -1 findbugs. The patch appears to cause Findbugs (version 2.0.3) to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in tajo-catalog/tajo-catalog-client tajo-catalog/tajo-catalog-common tajo-catalog/tajo-catalog-drivers/tajo-hive tajo-catalog/tajo-catalog-server tajo-common tajo-core tajo-core-tests tajo-plan: org.apache.tajo.engine.function.TestBuiltinFunctions org.apache.tajo.engine.query.TestCreateTable org.apache.tajo.engine.query.TestInnerJoinQuery org.apache.tajo.engine.function.TestConditionalExpressions org.apache.tajo.engine.query.TestInnerJoinWithSubQuery org.apache.tajo.engine.query.TestInsertQuery org.apache.tajo.engine.eval.TestSQLExpression org.apache.tajo.engine.query.TestNullValues org.apache.tajo.engine.query.TestJsonWithTimezone org.apache.tajo.engine.query.TestOuterJoinQuery org.apache.tajo.engine.query.TestCreateDatabase org.apache.tajo.util.history.TestHistoryWriterReader org.apache.tajo.engine.function.TestDateTimeFunctions org.apache.tajo.engine.query.TestWindowQuery org.apache.tajo.engine.query.TestOuterJoinWithSubQuery org.apache.tajo.master.TestNonForwardQueryResultSystemScanner org.apache.tajo.ws.rs.resources.TestQueryResultResource org.apache.tajo.engine.function.TestStringOperatorsAndFunctions org.apache.tajo.engine.eval.TestSQLDateTimeTypes org.apache.tajo.benchmark.TestTPCH org.apache.tajo.engine.planner.TestPlannerUtil org.apache.tajo.cli.tsql.commands.TestExecExternalShellCommand org.apache.tajo.ws.rs.resources.TestClusterResource org.apache.tajo.client.TestQueryClientExceptions org.apache.tajo.querymaster.TestQueryState org.apache.tajo.ws.rs.resources.TestTablesResource org.apache.tajo.engine.eval.TestPredicates org.apache.tajo.engine.query.TestAlterTablespace org.apache.tajo.ws.rs.resources.TestDatabasesResource org.apache.tajo.cli.tsql.TestTajoCli org.apache.tajo.engine.query.TestJoinOnPartitionedTables org.apache.tajo.engine.query.TestTruncateTable org.apache.tajo.engine.query.TestMultipleJoinTypes org.apache.tajo.engine.query.TestSelectQuery org.apache.tajo.engine.function.TestFunctionLoader org.apache.tajo.engine.codegen.TestEvalCodeGenerator org.apache.tajo.engine.query.TestInSubquery org.apache.tajo.cli.tsql.TestDefaultCliOutputFormatter org.apache.tajo.engine.function.TestMathFunctions org.apache.tajo.engine.query.TestNetTypes org.apache.tajo.engine.query.TestTablePartitions org.apache.tajo.engine.eval.TestIntervalType org.apache.tajo.master.TestQueryResult org.apache.tajo.engine.query.TestIndexScan org.apache.tajo.engine.planner.physical.TestSortExec org.apache.tajo.engine.query.TestGroupByQuery org.apache.tajo.engine.query.TestSetSessionQuery org.apache.tajo.engine.query.TestSelectNestedRecord org.apache.tajo.engine.function.TestJsonFunctions org.apache.tajo.cli.tools.TestTajoDump org.apache.tajo.engine.query.TestCreateIndex org.apache.tajo.engine.query.TestTableSubQuery org.apache.tajo.engine.planner.TestQueryValidation org.apache.tajo.engine.query.TestCommonConditionReduce org.apache.tajo.engine.query.TestDropTable org.apache.tajo.cli.tsql.commands.TestHdfsCommand org.apache.tajo.client.v2.TestTajoClientV2 org.apache.tajo.ws.rs.resources.TestFunctionsResource org.apache.tajo.engine.query.TestHBaseTable org.apache.tajo.engine.query.TestCrossJoin org.apache.tajo.querymaster.TestTaskStatusUpdate org.apache.tajo.querymaster.TestKillQuery org.apache.tajo.client.TestTajoClient org.apache.tajo.client.TestCatalogAdminClientExceptions org.apache.tajo.engine.query.TestSimpleQuery org.apache.tajo.ws.rs.resources.TestQueryResource org.apache.tajo.engine.function.TestPatternMatchingPredicates org.apache.tajo.engine.query.TestUnionQuery org.apache.tajo.cli.tsql.TestTajoCliNegatives org.apache.tajo.engine.function.TestPythonFunctions org.apache.tajo.querymaster.TestQueryProgress org.apache.tajo.engine.query.TestAlterTable org.apache.tajo.engine.query.TestCaseByCases org.apache.tajo.ws.rs.resources.TestSessionsResource org.apache.tajo.engine.eval.TestEvalTree org.apache.tajo.ha.TestHAServiceHDFSImpl org.apache.tajo.engine.query.TestSortQuery org.apache.tajo.engine.query.TestCTASQuery Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/861//testReport/ Findbugs results: https://builds.apache.org/job/PreCommit-TAJO-Build/861//findbugsResult Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/861//console This message is automatically generated.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/624#discussion_r39001856

          — Diff: tajo-catalog/tajo-catalog-client/src/main/java/org/apache/tajo/catalog/AbstractCatalogClient.java —
          @@ -446,6 +450,52 @@ public final PartitionDescProto getPartition(final String databaseName, final St
          }

          @Override
          + public boolean existPartitions(String databaseName, String tableName) {
          + try {
          + final BlockingInterface stub = getStub();
          + final TableIdentifierProto request = buildTableIdentifier(databaseName, tableName);
          + return isSuccess(stub.existPartitionsByTableName(null, request));
          — End diff –

          Even through this API is checking if the table exists, it potentially returns unexpected exception. If you check only its success, it may return a wrong result.

          So, you should return true or false depending on only both its success and UNDEFINED_PARTITION_METHODS. You can find the similar code pattern from ``existPartitionMethod`` method in this class.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/624#discussion_r39001856 — Diff: tajo-catalog/tajo-catalog-client/src/main/java/org/apache/tajo/catalog/AbstractCatalogClient.java — @@ -446,6 +450,52 @@ public final PartitionDescProto getPartition(final String databaseName, final St } @Override + public boolean existPartitions(String databaseName, String tableName) { + try { + final BlockingInterface stub = getStub(); + final TableIdentifierProto request = buildTableIdentifier(databaseName, tableName); + return isSuccess(stub.existPartitionsByTableName(null, request)); — End diff – Even through this API is checking if the table exists, it potentially returns unexpected exception. If you check only its success, it may return a wrong result. So, you should return true or false depending on only both its success and UNDEFINED_PARTITION_METHODS. You can find the similar code pattern from ``existPartitionMethod`` method in this class.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/624#discussion_r39002026

          — Diff: tajo-catalog/tajo-catalog-client/src/main/java/org/apache/tajo/catalog/AbstractCatalogClient.java —
          @@ -446,6 +450,52 @@ public final PartitionDescProto getPartition(final String databaseName, final St
          }

          @Override
          + public boolean existPartitions(String databaseName, String tableName) {
          + try

          { + final BlockingInterface stub = getStub(); + final TableIdentifierProto request = buildTableIdentifier(databaseName, tableName); + return isSuccess(stub.existPartitionsByTableName(null, request)); + }

          catch (ServiceException e)

          { + throw new RuntimeException(e); + }

          + }
          +
          + @Override
          + public List<PartitionDescProto> getPartitionsByAlgebra(PartitionsByAlgebraProto request) throws
          + UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException {
          + try {
          + final BlockingInterface stub = getStub();
          + GetPartitionsResponse response = stub.getPartitionsByAlgebra(null, request);
          +
          + throwsIfThisError(response.getState(), UndefinedDatabaseException.class);
          + throwsIfThisError(response.getState(), UndefinedTableException.class);
          + throwsIfThisError(response.getState(), UndefinedPartitionMethodException.class);
          + ensureOk(response.getState());
          + return response.getPartitionList();
          — End diff –

          mis indent.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/624#discussion_r39002026 — Diff: tajo-catalog/tajo-catalog-client/src/main/java/org/apache/tajo/catalog/AbstractCatalogClient.java — @@ -446,6 +450,52 @@ public final PartitionDescProto getPartition(final String databaseName, final St } @Override + public boolean existPartitions(String databaseName, String tableName) { + try { + final BlockingInterface stub = getStub(); + final TableIdentifierProto request = buildTableIdentifier(databaseName, tableName); + return isSuccess(stub.existPartitionsByTableName(null, request)); + } catch (ServiceException e) { + throw new RuntimeException(e); + } + } + + @Override + public List<PartitionDescProto> getPartitionsByAlgebra(PartitionsByAlgebraProto request) throws + UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException { + try { + final BlockingInterface stub = getStub(); + GetPartitionsResponse response = stub.getPartitionsByAlgebra(null, request); + + throwsIfThisError(response.getState(), UndefinedDatabaseException.class); + throwsIfThisError(response.getState(), UndefinedTableException.class); + throwsIfThisError(response.getState(), UndefinedPartitionMethodException.class); + ensureOk(response.getState()); + return response.getPartitionList(); — End diff – mis indent.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/624#discussion_r39007546

          — Diff: tajo-catalog/tajo-catalog-client/src/main/java/org/apache/tajo/catalog/AbstractCatalogClient.java —
          @@ -446,6 +450,52 @@ public final PartitionDescProto getPartition(final String databaseName, final St
          }

          @Override
          + public boolean existPartitions(String databaseName, String tableName) {
          — End diff –

          How about renaming it into ``isPartitionedTable``? It seems to be better.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/624#discussion_r39007546 — Diff: tajo-catalog/tajo-catalog-client/src/main/java/org/apache/tajo/catalog/AbstractCatalogClient.java — @@ -446,6 +450,52 @@ public final PartitionDescProto getPartition(final String databaseName, final St } @Override + public boolean existPartitions(String databaseName, String tableName) { — End diff – How about renaming it into ``isPartitionedTable``? It seems to be better.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/624#discussion_r39007654

          — Diff: tajo-catalog/tajo-catalog-client/src/main/java/org/apache/tajo/catalog/AbstractCatalogClient.java —
          @@ -428,15 +429,18 @@ public final PartitionDescProto getPartition(final String databaseName, final St
          }

          @Override

          • public final List<PartitionDescProto> getPartitions(final String databaseName, final String tableName) {
            + public final List<PartitionDescProto> getAllPartitions(final String databaseName, final String tableName) throws
            + UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException,
            + UndefinedPartitionException {
              • End diff –

          It retrieves all partitions. So it may be inproper to throw ``UndefinedPartitionException``, which should be thrown in the case where there is no partition corresponding to a specific partition key.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/624#discussion_r39007654 — Diff: tajo-catalog/tajo-catalog-client/src/main/java/org/apache/tajo/catalog/AbstractCatalogClient.java — @@ -428,15 +429,18 @@ public final PartitionDescProto getPartition(final String databaseName, final St } @Override public final List<PartitionDescProto> getPartitions(final String databaseName, final String tableName) { + public final List<PartitionDescProto> getAllPartitions(final String databaseName, final String tableName) throws + UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException, + UndefinedPartitionException { End diff – It retrieves all partitions. So it may be inproper to throw ``UndefinedPartitionException``, which should be thrown in the case where there is no partition corresponding to a specific partition key.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/624#discussion_r39007741

          — Diff: tajo-catalog/tajo-catalog-client/src/main/proto/CatalogProtocol.proto —
          @@ -121,9 +121,12 @@ service CatalogProtocolService {
          rpc existPartitionMethod(TableIdentifierProto) returns (ReturnState);

          rpc getPartitionByPartitionName(PartitionIdentifierProto) returns (GetPartitionDescResponse);

          • rpc getPartitionsByTableName(PartitionIdentifierProto) returns (GetPartitionsResponse);
            + rpc getPartitionsByTableName(TableIdentifierProto) returns (GetPartitionsResponse);
            + rpc existPartitionsByTableName(TableIdentifierProto) returns (ReturnState);
              • End diff –

          As I mentioned, ``isPartitionedTable`` would be better. It may not need the suffix ``byPartitionName``.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/624#discussion_r39007741 — Diff: tajo-catalog/tajo-catalog-client/src/main/proto/CatalogProtocol.proto — @@ -121,9 +121,12 @@ service CatalogProtocolService { rpc existPartitionMethod(TableIdentifierProto) returns (ReturnState); rpc getPartitionByPartitionName(PartitionIdentifierProto) returns (GetPartitionDescResponse); rpc getPartitionsByTableName(PartitionIdentifierProto) returns (GetPartitionsResponse); + rpc getPartitionsByTableName(TableIdentifierProto) returns (GetPartitionsResponse); + rpc existPartitionsByTableName(TableIdentifierProto) returns (ReturnState); End diff – As I mentioned, ``isPartitionedTable`` would be better. It may not need the suffix ``byPartitionName``.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/624#discussion_r39008058

          — Diff: tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogService.java —
          @@ -162,7 +162,17 @@ PartitionDescProto getPartition(String databaseName, String tableName, String pa
          throws UndefinedPartitionException, UndefinedPartitionMethodException, UndefinedDatabaseException,
          UndefinedTableException;

          • List<PartitionDescProto> getPartitions(String databaseName, String tableName);
            + List<PartitionDescProto> getAllPartitions(String databaseName, String tableName) throws UndefinedDatabaseException,
            + UndefinedTableException, UndefinedPartitionMethodException, UndefinedPartitionException, UndefinedOperatorException;
            +
            + boolean existPartitions(String databaseName, String tableName) throws UndefinedDatabaseException,
            + UndefinedTableException, UndefinedPartitionMethodException;
            +
            + List<PartitionDescProto> getPartitionsByAlgebra(PartitionsByAlgebraProto request) throws
            + UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException;
            +
            + List<PartitionDescProto> getPartitionsByDirectSql(PartitionsByDirectSqlProto request) throws
              • End diff –

          Its name should be changed into PartitionByFilter or PartitionByPredicate. As far as I know, the suffix ``DirectSQL`` is inspired by Hive API. But, the Hive API does predefined filter expression instead of direct sql predicate. Direct SQL is always danger, and this is not our way.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/624#discussion_r39008058 — Diff: tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogService.java — @@ -162,7 +162,17 @@ PartitionDescProto getPartition(String databaseName, String tableName, String pa throws UndefinedPartitionException, UndefinedPartitionMethodException, UndefinedDatabaseException, UndefinedTableException; List<PartitionDescProto> getPartitions(String databaseName, String tableName); + List<PartitionDescProto> getAllPartitions(String databaseName, String tableName) throws UndefinedDatabaseException, + UndefinedTableException, UndefinedPartitionMethodException, UndefinedPartitionException, UndefinedOperatorException; + + boolean existPartitions(String databaseName, String tableName) throws UndefinedDatabaseException, + UndefinedTableException, UndefinedPartitionMethodException; + + List<PartitionDescProto> getPartitionsByAlgebra(PartitionsByAlgebraProto request) throws + UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException; + + List<PartitionDescProto> getPartitionsByDirectSql(PartitionsByDirectSqlProto request) throws End diff – Its name should be changed into PartitionByFilter or PartitionByPredicate . As far as I know, the suffix ``DirectSQL`` is inspired by Hive API. But, the Hive API does predefined filter expression instead of direct sql predicate. Direct SQL is always danger, and this is not our way.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/624#discussion_r39008262

          — Diff: tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogStore.java —
          @@ -845,11 +854,72 @@ public boolean existPartitionMethod(String databaseName, String tableName) throw
          }

          @Override

          • public List<CatalogProtos.PartitionDescProto> getPartitions(String databaseName,
          • String tableName) {
          • throw new UnsupportedOperationException();
            + public List<CatalogProtos.PartitionDescProto> getPartitions(String databaseName, String tableName) {
            + throw new TajoRuntimeException(new UnsupportedException("getPartitions"));
              • End diff –

          UnsupportedException automatically generates the error message including the method and class if it is used with a default constructor. SO, it should be ``new UnsupportedException()``.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/624#discussion_r39008262 — Diff: tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogStore.java — @@ -845,11 +854,72 @@ public boolean existPartitionMethod(String databaseName, String tableName) throw } @Override public List<CatalogProtos.PartitionDescProto> getPartitions(String databaseName, String tableName) { throw new UnsupportedOperationException(); + public List<CatalogProtos.PartitionDescProto> getPartitions(String databaseName, String tableName) { + throw new TajoRuntimeException(new UnsupportedException("getPartitions")); End diff – UnsupportedException automatically generates the error message including the method and class if it is used with a default constructor. SO, it should be ``new UnsupportedException()``.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/624#discussion_r39008270

          — Diff: tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogStore.java —
          @@ -845,11 +854,72 @@ public boolean existPartitionMethod(String databaseName, String tableName) throw
          }

          @Override

          • public List<CatalogProtos.PartitionDescProto> getPartitions(String databaseName,
          • String tableName) {
          • throw new UnsupportedOperationException();
            + public List<CatalogProtos.PartitionDescProto> getPartitions(String databaseName, String tableName) { + throw new TajoRuntimeException(new UnsupportedException("getPartitions")); + }

            +
            + @Override
            + public boolean existPartitions(String databaseName, String tableName)

            { + throw new TajoRuntimeException(new UnsupportedException("existPartitions")); + }

            +
            + @Override
            + public List<PartitionDescProto> getPartitionsByAlgebra(PartitionsByAlgebraProto request) {
            + throw new TajoRuntimeException(new UnsupportedException("getPartitionsByAlgebra"));

              • End diff –

          UnsupportedException automatically generates the error message including the method and class if it is used with a default constructor. SO, it should be new UnsupportedException().

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/624#discussion_r39008270 — Diff: tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogStore.java — @@ -845,11 +854,72 @@ public boolean existPartitionMethod(String databaseName, String tableName) throw } @Override public List<CatalogProtos.PartitionDescProto> getPartitions(String databaseName, String tableName) { throw new UnsupportedOperationException(); + public List<CatalogProtos.PartitionDescProto> getPartitions(String databaseName, String tableName) { + throw new TajoRuntimeException(new UnsupportedException("getPartitions")); + } + + @Override + public boolean existPartitions(String databaseName, String tableName) { + throw new TajoRuntimeException(new UnsupportedException("existPartitions")); + } + + @Override + public List<PartitionDescProto> getPartitionsByAlgebra(PartitionsByAlgebraProto request) { + throw new TajoRuntimeException(new UnsupportedException("getPartitionsByAlgebra")); End diff – UnsupportedException automatically generates the error message including the method and class if it is used with a default constructor. SO, it should be new UnsupportedException().
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/624#discussion_r39008520

          — Diff: tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogStore.java —
          @@ -1052,7 +1122,7 @@ public void addPartitions(String databaseName, String tableName, List<CatalogPro
          }

          if (addPartitions.size() > 0) {

          • client.getHiveClient().add_partitions(addPartitions, true, true);
            + List<Partition> results = client.getHiveClient().add_partitions(addPartitions, true, true);
              • End diff –

          ``List<Partition> results`` is not used.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/624#discussion_r39008520 — Diff: tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogStore.java — @@ -1052,7 +1122,7 @@ public void addPartitions(String databaseName, String tableName, List<CatalogPro } if (addPartitions.size() > 0) { client.getHiveClient().add_partitions(addPartitions, true, true); + List<Partition> results = client.getHiveClient().add_partitions(addPartitions, true, true); End diff – ``List<Partition> results`` is not used.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/624#discussion_r39009348

          — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java —
          @@ -2173,6 +2179,248 @@ private void setPartitionKeys(int pid, PartitionDescProto.Builder partitionDesc)
          }

          @Override
          + public boolean existPartitions(String databaseName, String tableName) throws UndefinedDatabaseException,
          — End diff –

          It would be better if it is renamed into ``isPartitionedTable``.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/624#discussion_r39009348 — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java — @@ -2173,6 +2179,248 @@ private void setPartitionKeys(int pid, PartitionDescProto.Builder partitionDesc) } @Override + public boolean existPartitions(String databaseName, String tableName) throws UndefinedDatabaseException, — End diff – It would be better if it is renamed into ``isPartitionedTable``.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/624#discussion_r39009365

          — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java —
          @@ -2132,7 +2137,8 @@ private void setPartitionKeys(int pid, PartitionDescProto.Builder partitionDesc)

          @Override
          public List<PartitionDescProto> getPartitions(String databaseName, String tableName)
          — End diff –

          It should be getAllPartitions for naming consistency.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/624#discussion_r39009365 — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java — @@ -2132,7 +2137,8 @@ private void setPartitionKeys(int pid, PartitionDescProto.Builder partitionDesc) @Override public List<PartitionDescProto> getPartitions(String databaseName, String tableName) — End diff – It should be getAllPartitions for naming consistency.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/624#discussion_r39010692

          — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java —
          @@ -2173,6 +2179,248 @@ private void setPartitionKeys(int pid, PartitionDescProto.Builder partitionDesc)
          }

          @Override
          + public boolean existPartitions(String databaseName, String tableName) throws UndefinedDatabaseException,
          + UndefinedTableException, UndefinedPartitionMethodException {
          + Connection conn = null;
          + ResultSet res = null;
          + PreparedStatement pstmt = null;
          + boolean result = false;
          +
          + try {
          + int databaseId = getDatabaseId(databaseName);
          + int tableId = getTableId(databaseId, databaseName, tableName);
          +
          + if (!existPartitionMethod(databaseName, tableName))

          { + throw new UndefinedPartitionMethodException(tableName); + }

          +
          + String sql = "SELECT COUNT CNT FROM "
          + + TB_PARTTIONS +" WHERE " + COL_TABLES_PK + " = ? ";
          +
          + if (LOG.isDebugEnabled())

          { + LOG.debug(sql); + }

          +
          + conn = getConnection();
          + pstmt = conn.prepareStatement(sql);
          + pstmt.setInt(1, tableId);
          + res = pstmt.executeQuery();
          +
          + if (res.next()) {
          + if (res.getInt("CNT") > 0)

          { + result = true; + }

          + }
          + } catch (SQLException se)

          { + throw new TajoInternalError(se); + }

          finally

          { + CatalogUtil.closeQuietly(pstmt, res); + }

          + return result;
          + }
          +
          + @Override
          + public List<PartitionDescProto> getPartitionsByDirectSql(PartitionsByDirectSqlProto request)

          { + throw new TajoRuntimeException(new UnsupportedException("getPartitionsByDirectSql")); + }

          +
          + @Override
          + public List<PartitionDescProto> getPartitionsByAlgebra(PartitionsByAlgebraProto request) throws
          + UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException {
          + Connection conn = null;
          + PreparedStatement pstmt = null;
          + ResultSet res = null;
          + int currentIndex = 1;
          + String directSQL = null;
          +
          + List<PartitionDescProto> partitions = TUtil.newList();
          + List<PartitionFilterSet> filterSets = TUtil.newList();
          +
          + try {
          + int databaseId = getDatabaseId(request.getDatabaseName());
          + int tableId = getTableId(databaseId, request.getDatabaseName(), request.getTableName());
          + if (!existPartitionMethod(request.getDatabaseName(), request.getTableName()))

          { + throw new UndefinedPartitionMethodException(request.getTableName()); + }

          +
          + TableDescProto tableDesc = getTable(request.getDatabaseName(), request.getTableName());
          + conn = getConnection();
          — End diff –

          It may be very trivial. It would be better if this line is moved above ``pstmt = conn.prepareStatement(directSQL);``.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/624#discussion_r39010692 — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java — @@ -2173,6 +2179,248 @@ private void setPartitionKeys(int pid, PartitionDescProto.Builder partitionDesc) } @Override + public boolean existPartitions(String databaseName, String tableName) throws UndefinedDatabaseException, + UndefinedTableException, UndefinedPartitionMethodException { + Connection conn = null; + ResultSet res = null; + PreparedStatement pstmt = null; + boolean result = false; + + try { + int databaseId = getDatabaseId(databaseName); + int tableId = getTableId(databaseId, databaseName, tableName); + + if (!existPartitionMethod(databaseName, tableName)) { + throw new UndefinedPartitionMethodException(tableName); + } + + String sql = "SELECT COUNT CNT FROM " + + TB_PARTTIONS +" WHERE " + COL_TABLES_PK + " = ? "; + + if (LOG.isDebugEnabled()) { + LOG.debug(sql); + } + + conn = getConnection(); + pstmt = conn.prepareStatement(sql); + pstmt.setInt(1, tableId); + res = pstmt.executeQuery(); + + if (res.next()) { + if (res.getInt("CNT") > 0) { + result = true; + } + } + } catch (SQLException se) { + throw new TajoInternalError(se); + } finally { + CatalogUtil.closeQuietly(pstmt, res); + } + return result; + } + + @Override + public List<PartitionDescProto> getPartitionsByDirectSql(PartitionsByDirectSqlProto request) { + throw new TajoRuntimeException(new UnsupportedException("getPartitionsByDirectSql")); + } + + @Override + public List<PartitionDescProto> getPartitionsByAlgebra(PartitionsByAlgebraProto request) throws + UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException { + Connection conn = null; + PreparedStatement pstmt = null; + ResultSet res = null; + int currentIndex = 1; + String directSQL = null; + + List<PartitionDescProto> partitions = TUtil.newList(); + List<PartitionFilterSet> filterSets = TUtil.newList(); + + try { + int databaseId = getDatabaseId(request.getDatabaseName()); + int tableId = getTableId(databaseId, request.getDatabaseName(), request.getTableName()); + if (!existPartitionMethod(request.getDatabaseName(), request.getTableName())) { + throw new UndefinedPartitionMethodException(request.getTableName()); + } + + TableDescProto tableDesc = getTable(request.getDatabaseName(), request.getTableName()); + conn = getConnection(); — End diff – It may be very trivial. It would be better if this line is moved above ``pstmt = conn.prepareStatement(directSQL);``.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/624#discussion_r39012228

          — Diff: tajo-plan/src/main/java/org/apache/tajo/plan/rewrite/rules/PartitionedTableRewriter.java —
          @@ -118,32 +121,264 @@ public String toString() {

          • @return
          • @throws IOException
            */
          • private Path [] findFilteredPaths(OverridableConf queryContext, Schema partitionColumns, EvalNode [] conjunctiveForms,
          • Path tablePath)
          • throws IOException {
            + private Path [] findFilteredPaths(OverridableConf queryContext, String tableName,
            + Schema partitionColumns, EvalNode [] conjunctiveForms, Path tablePath)
            + throws IOException, UndefinedDatabaseException, UndefinedTableException,
            + UndefinedPartitionMethodException, UndefinedOperatorException {

          + Path [] filteredPaths = null;
          FileSystem fs = tablePath.getFileSystem(queryContext.getConf());
          + String [] splits = CatalogUtil.splitFQTableName(tableName);
          + List<PartitionDescProto> partitions = null;

          • PathFilter [] filters;
          • if (conjunctiveForms == null) { - filters = buildAllAcceptingPathFilters(partitionColumns); - }

            else {

          • filters = buildPathFiltersForAllLevels(partitionColumns, conjunctiveForms);
            + String store = queryContext.getConf().get(CatalogConstants.STORE_CLASS);
            +
            + try {
            + // HiveCatalogStore provides list of table partitions with where clause because hive just provides api using
            + // the filter string. So, this rewriter need to differentiate HiveCatalogStore and other catalogs.
            + if (store.equals("org.apache.tajo.catalog.store.HiveCatalogStore")) {
              • End diff –

          A RewriteRule should not consider its catalog storage type. It does not make sense. CatalogService should just take an algebra, and HiveCatalogStore should generates a filter expression string representation from the algebra.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/624#discussion_r39012228 — Diff: tajo-plan/src/main/java/org/apache/tajo/plan/rewrite/rules/PartitionedTableRewriter.java — @@ -118,32 +121,264 @@ public String toString() { @return @throws IOException */ private Path [] findFilteredPaths(OverridableConf queryContext, Schema partitionColumns, EvalNode [] conjunctiveForms, Path tablePath) throws IOException { + private Path [] findFilteredPaths(OverridableConf queryContext, String tableName, + Schema partitionColumns, EvalNode [] conjunctiveForms, Path tablePath) + throws IOException, UndefinedDatabaseException, UndefinedTableException, + UndefinedPartitionMethodException, UndefinedOperatorException { + Path [] filteredPaths = null; FileSystem fs = tablePath.getFileSystem(queryContext.getConf()); + String [] splits = CatalogUtil.splitFQTableName(tableName); + List<PartitionDescProto> partitions = null; PathFilter [] filters; if (conjunctiveForms == null) { - filters = buildAllAcceptingPathFilters(partitionColumns); - } else { filters = buildPathFiltersForAllLevels(partitionColumns, conjunctiveForms); + String store = queryContext.getConf().get(CatalogConstants.STORE_CLASS); + + try { + // HiveCatalogStore provides list of table partitions with where clause because hive just provides api using + // the filter string. So, this rewriter need to differentiate HiveCatalogStore and other catalogs. + if (store.equals("org.apache.tajo.catalog.store.HiveCatalogStore")) { End diff – A RewriteRule should not consider its catalog storage type. It does not make sense. CatalogService should just take an algebra, and HiveCatalogStore should generates a filter expression string representation from the algebra.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user blrunner commented on the pull request:

          https://github.com/apache/tajo/pull/624#issuecomment-139205154

          @hyunsik

          Thank you for your detailed review and I've just reflected your comments.

          Show
          githubbot ASF GitHub Bot added a comment - Github user blrunner commented on the pull request: https://github.com/apache/tajo/pull/624#issuecomment-139205154 @hyunsik Thank you for your detailed review and I've just reflected your comments.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/624#discussion_r39270682

          — Diff: tajo-common/src/main/java/org/apache/tajo/exception/ReturnStateUtil.java —
          @@ -161,6 +161,10 @@ public static ReturnState errUndefinedPartition(String partitionName)

          { return returnError(ResultCode.UNDEFINED_PARTITION, partitionName); }

          + public static ReturnState errPartitionNotFund(String tbName) {
          — End diff –

          If it is not used, you may do not need to add this method.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/624#discussion_r39270682 — Diff: tajo-common/src/main/java/org/apache/tajo/exception/ReturnStateUtil.java — @@ -161,6 +161,10 @@ public static ReturnState errUndefinedPartition(String partitionName) { return returnError(ResultCode.UNDEFINED_PARTITION, partitionName); } + public static ReturnState errPartitionNotFund(String tbName) { — End diff – If it is not used, you may do not need to add this method.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/624#discussion_r39270987

          — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java —
          @@ -2172,6 +2178,252 @@ private void setPartitionKeys(int pid, PartitionDescProto.Builder partitionDesc)
          return partitions;
          }

          + /**
          + * Check if list of partitios exist on catalog.
          — End diff –

          ``partitios`` seems to be a typo error.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/624#discussion_r39270987 — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java — @@ -2172,6 +2178,252 @@ private void setPartitionKeys(int pid, PartitionDescProto.Builder partitionDesc) return partitions; } + /** + * Check if list of partitios exist on catalog. — End diff – ``partitios`` seems to be a typo error.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/624#discussion_r39271326

          — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java —
          @@ -2172,6 +2178,252 @@ private void setPartitionKeys(int pid, PartitionDescProto.Builder partitionDesc)
          return partitions;
          }

          + /**
          + * Check if list of partitios exist on catalog.
          + *
          + *
          + * @param databaseId
          + * @param tableId
          + * @return
          + */
          + public boolean existPartitionsOnCatalog(int databaseId, int tableId) {
          + Connection conn = null;
          + ResultSet res = null;
          + PreparedStatement pstmt = null;
          + boolean result = false;
          +
          + try {
          + String sql = "SELECT COUNT CNT FROM "
          + + TB_PARTTIONS +" WHERE " + COL_TABLES_PK + " = ? ";
          +
          + if (LOG.isDebugEnabled())

          { + LOG.debug(sql); + }

          +
          + conn = getConnection();
          + pstmt = conn.prepareStatement(sql);
          + pstmt.setInt(1, tableId);
          + res = pstmt.executeQuery();
          +
          + if (res.next()) {
          + if (res.getInt("CNT") > 0)

          { + result = true; + }

          + }
          + } catch (SQLException se)

          { + throw new TajoInternalError(se); + }

          finally

          { + CatalogUtil.closeQuietly(pstmt, res); + }

          + return result;
          + }
          +
          + @Override
          + public List<PartitionDescProto> getPartitionsByFilter(PartitionsByFilterProto request)

          { + throw new TajoRuntimeException(new UnsupportedException()); + }

          +
          + @Override
          + public List<PartitionDescProto> getPartitionsByAlgebra(PartitionsByAlgebraProto request) throws
          + UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException {
          + Connection conn = null;
          + PreparedStatement pstmt = null;
          + ResultSet res = null;
          + int currentIndex = 1;
          + String selectStatement = null;
          +
          + List<PartitionDescProto> partitions = TUtil.newList();
          + List<PartitionFilterSet> filterSets = TUtil.newList();
          +
          + try {
          + int databaseId = getDatabaseId(request.getDatabaseName());
          + int tableId = getTableId(databaseId, request.getDatabaseName(), request.getTableName());
          + if (!existPartitionMethod(request.getDatabaseName(), request.getTableName()))

          { + throw new UndefinedPartitionMethodException(request.getTableName()); + }

          +
          + if (!existPartitionsOnCatalog(databaseId, tableId)) {
          + throw new PartitionNotFoundException(request.getTableName());
          — End diff –

          Could you elaborate the purpose of this exception? Does it means that there is no partition? If so, it does not make sense because it returns a list of partitions. So, it should return an empty list if there is no partitions.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/624#discussion_r39271326 — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java — @@ -2172,6 +2178,252 @@ private void setPartitionKeys(int pid, PartitionDescProto.Builder partitionDesc) return partitions; } + /** + * Check if list of partitios exist on catalog. + * + * + * @param databaseId + * @param tableId + * @return + */ + public boolean existPartitionsOnCatalog(int databaseId, int tableId) { + Connection conn = null; + ResultSet res = null; + PreparedStatement pstmt = null; + boolean result = false; + + try { + String sql = "SELECT COUNT CNT FROM " + + TB_PARTTIONS +" WHERE " + COL_TABLES_PK + " = ? "; + + if (LOG.isDebugEnabled()) { + LOG.debug(sql); + } + + conn = getConnection(); + pstmt = conn.prepareStatement(sql); + pstmt.setInt(1, tableId); + res = pstmt.executeQuery(); + + if (res.next()) { + if (res.getInt("CNT") > 0) { + result = true; + } + } + } catch (SQLException se) { + throw new TajoInternalError(se); + } finally { + CatalogUtil.closeQuietly(pstmt, res); + } + return result; + } + + @Override + public List<PartitionDescProto> getPartitionsByFilter(PartitionsByFilterProto request) { + throw new TajoRuntimeException(new UnsupportedException()); + } + + @Override + public List<PartitionDescProto> getPartitionsByAlgebra(PartitionsByAlgebraProto request) throws + UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException { + Connection conn = null; + PreparedStatement pstmt = null; + ResultSet res = null; + int currentIndex = 1; + String selectStatement = null; + + List<PartitionDescProto> partitions = TUtil.newList(); + List<PartitionFilterSet> filterSets = TUtil.newList(); + + try { + int databaseId = getDatabaseId(request.getDatabaseName()); + int tableId = getTableId(databaseId, request.getDatabaseName(), request.getTableName()); + if (!existPartitionMethod(request.getDatabaseName(), request.getTableName())) { + throw new UndefinedPartitionMethodException(request.getTableName()); + } + + if (!existPartitionsOnCatalog(databaseId, tableId)) { + throw new PartitionNotFoundException(request.getTableName()); — End diff – Could you elaborate the purpose of this exception? Does it means that there is no partition? If so, it does not make sense because it returns a list of partitions. So, it should return an empty list if there is no partitions.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/624#discussion_r39271506

          — Diff: tajo-catalog/tajo-catalog-client/src/main/java/org/apache/tajo/catalog/AbstractCatalogClient.java —
          @@ -25,6 +25,7 @@
          import org.apache.tajo.catalog.CatalogProtocol.CatalogProtocolService.BlockingInterface;
          import org.apache.tajo.catalog.CatalogProtocol.*;
          import org.apache.tajo.catalog.partition.PartitionMethodDesc;
          +import org.apache.tajo.catalog.proto.CatalogProtos;
          — End diff –

          It seems to be not used.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/624#discussion_r39271506 — Diff: tajo-catalog/tajo-catalog-client/src/main/java/org/apache/tajo/catalog/AbstractCatalogClient.java — @@ -25,6 +25,7 @@ import org.apache.tajo.catalog.CatalogProtocol.CatalogProtocolService.BlockingInterface; import org.apache.tajo.catalog.CatalogProtocol.*; import org.apache.tajo.catalog.partition.PartitionMethodDesc; +import org.apache.tajo.catalog.proto.CatalogProtos; — End diff – It seems to be not used.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/624#discussion_r39271781

          — Diff: tajo-plan/src/main/java/org/apache/tajo/plan/util/ScanQualConverter.java —
          @@ -0,0 +1,297 @@
          +/**
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +
          +package org.apache.tajo.plan.util;
          +
          +import org.apache.tajo.algebra.*;
          +import org.apache.tajo.datum.DateDatum;
          +import org.apache.tajo.datum.Datum;
          +import org.apache.tajo.datum.TimeDatum;
          +import org.apache.tajo.datum.TimestampDatum;
          +import org.apache.tajo.plan.expr.*;
          +
          +import java.util.Stack;
          +
          +/**
          + * This converts Quals of ScanNode to Algebra expressions.
          + *
          + */
          +public class ScanQualConverter extends SimpleEvalNodeVisitor<Object> {
          — End diff –

          I'd like to recommend renaming it to ``EvalNodeToExprConverter``. It appears to be more general in terms of converting any EvalNode tree into Expr tree.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/624#discussion_r39271781 — Diff: tajo-plan/src/main/java/org/apache/tajo/plan/util/ScanQualConverter.java — @@ -0,0 +1,297 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.tajo.plan.util; + +import org.apache.tajo.algebra.*; +import org.apache.tajo.datum.DateDatum; +import org.apache.tajo.datum.Datum; +import org.apache.tajo.datum.TimeDatum; +import org.apache.tajo.datum.TimestampDatum; +import org.apache.tajo.plan.expr.*; + +import java.util.Stack; + +/** + * This converts Quals of ScanNode to Algebra expressions. + * + */ +public class ScanQualConverter extends SimpleEvalNodeVisitor<Object> { — End diff – I'd like to recommend renaming it to ``EvalNodeToExprConverter``. It appears to be more general in terms of converting any EvalNode tree into Expr tree.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user blrunner commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/624#discussion_r39335162

          — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java —
          @@ -2172,6 +2178,252 @@ private void setPartitionKeys(int pid, PartitionDescProto.Builder partitionDesc)
          return partitions;
          }

          + /**
          + * Check if list of partitios exist on catalog.
          + *
          + *
          + * @param databaseId
          + * @param tableId
          + * @return
          + */
          + public boolean existPartitionsOnCatalog(int databaseId, int tableId) {
          + Connection conn = null;
          + ResultSet res = null;
          + PreparedStatement pstmt = null;
          + boolean result = false;
          +
          + try {
          + String sql = "SELECT COUNT CNT FROM "
          + + TB_PARTTIONS +" WHERE " + COL_TABLES_PK + " = ? ";
          +
          + if (LOG.isDebugEnabled())

          { + LOG.debug(sql); + }

          +
          + conn = getConnection();
          + pstmt = conn.prepareStatement(sql);
          + pstmt.setInt(1, tableId);
          + res = pstmt.executeQuery();
          +
          + if (res.next()) {
          + if (res.getInt("CNT") > 0)

          { + result = true; + }

          + }
          + } catch (SQLException se)

          { + throw new TajoInternalError(se); + }

          finally

          { + CatalogUtil.closeQuietly(pstmt, res); + }

          + return result;
          + }
          +
          + @Override
          + public List<PartitionDescProto> getPartitionsByFilter(PartitionsByFilterProto request)

          { + throw new TajoRuntimeException(new UnsupportedException()); + }

          +
          + @Override
          + public List<PartitionDescProto> getPartitionsByAlgebra(PartitionsByAlgebraProto request) throws
          + UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException {
          + Connection conn = null;
          + PreparedStatement pstmt = null;
          + ResultSet res = null;
          + int currentIndex = 1;
          + String selectStatement = null;
          +
          + List<PartitionDescProto> partitions = TUtil.newList();
          + List<PartitionFilterSet> filterSets = TUtil.newList();
          +
          + try {
          + int databaseId = getDatabaseId(request.getDatabaseName());
          + int tableId = getTableId(databaseId, request.getDatabaseName(), request.getTableName());
          + if (!existPartitionMethod(request.getDatabaseName(), request.getTableName()))

          { + throw new UndefinedPartitionMethodException(request.getTableName()); + }

          +
          + if (!existPartitionsOnCatalog(databaseId, tableId)) {
          + throw new PartitionNotFoundException(request.getTableName());
          — End diff –

          The purpose is for partition pruning when there is no partitions on catalog and partition directories exist on file system. Above case, users need to execute recover partition (https://github.com/apache/tajo/pull/626). But we need to handle their queries if they don't recover their partitions located on file system to catalog. So, I implemented the exception and if catalog throws the exception, PartitionedTableRewriter would set a list of partitions to null. And then, the rewriter would check directories from file system.

          Show
          githubbot ASF GitHub Bot added a comment - Github user blrunner commented on a diff in the pull request: https://github.com/apache/tajo/pull/624#discussion_r39335162 — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java — @@ -2172,6 +2178,252 @@ private void setPartitionKeys(int pid, PartitionDescProto.Builder partitionDesc) return partitions; } + /** + * Check if list of partitios exist on catalog. + * + * + * @param databaseId + * @param tableId + * @return + */ + public boolean existPartitionsOnCatalog(int databaseId, int tableId) { + Connection conn = null; + ResultSet res = null; + PreparedStatement pstmt = null; + boolean result = false; + + try { + String sql = "SELECT COUNT CNT FROM " + + TB_PARTTIONS +" WHERE " + COL_TABLES_PK + " = ? "; + + if (LOG.isDebugEnabled()) { + LOG.debug(sql); + } + + conn = getConnection(); + pstmt = conn.prepareStatement(sql); + pstmt.setInt(1, tableId); + res = pstmt.executeQuery(); + + if (res.next()) { + if (res.getInt("CNT") > 0) { + result = true; + } + } + } catch (SQLException se) { + throw new TajoInternalError(se); + } finally { + CatalogUtil.closeQuietly(pstmt, res); + } + return result; + } + + @Override + public List<PartitionDescProto> getPartitionsByFilter(PartitionsByFilterProto request) { + throw new TajoRuntimeException(new UnsupportedException()); + } + + @Override + public List<PartitionDescProto> getPartitionsByAlgebra(PartitionsByAlgebraProto request) throws + UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException { + Connection conn = null; + PreparedStatement pstmt = null; + ResultSet res = null; + int currentIndex = 1; + String selectStatement = null; + + List<PartitionDescProto> partitions = TUtil.newList(); + List<PartitionFilterSet> filterSets = TUtil.newList(); + + try { + int databaseId = getDatabaseId(request.getDatabaseName()); + int tableId = getTableId(databaseId, request.getDatabaseName(), request.getTableName()); + if (!existPartitionMethod(request.getDatabaseName(), request.getTableName())) { + throw new UndefinedPartitionMethodException(request.getTableName()); + } + + if (!existPartitionsOnCatalog(databaseId, tableId)) { + throw new PartitionNotFoundException(request.getTableName()); — End diff – The purpose is for partition pruning when there is no partitions on catalog and partition directories exist on file system. Above case, users need to execute recover partition ( https://github.com/apache/tajo/pull/626 ). But we need to handle their queries if they don't recover their partitions located on file system to catalog. So, I implemented the exception and if catalog throws the exception, PartitionedTableRewriter would set a list of partitions to null. And then, the rewriter would check directories from file system.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user blrunner commented on the pull request:

          https://github.com/apache/tajo/pull/624#issuecomment-139737937

          Thanks @hyunsik

          I reflected your comments and left some comments.

          Show
          githubbot ASF GitHub Bot added a comment - Github user blrunner commented on the pull request: https://github.com/apache/tajo/pull/624#issuecomment-139737937 Thanks @hyunsik I reflected your comments and left some comments.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/624#discussion_r39639478

          — Diff: tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestTablePartitions.java —
          @@ -1325,4 +1332,421 @@ public final void testDuplicatedPartitions() throws Exception

          { executeString("DROP TABLE " + tableName + " PURGE"); }

          }
          +
          + @Test
          + public final void testPatternMatchingPredicatesAndStringFunctions() throws Exception {
          + ResultSet res = null;
          + String tableName = CatalogUtil.normalizeIdentifier("testPatternMatchingPredicatesAndStringFunctions");
          + String expectedResult;
          +
          + if (nodeType == NodeType.INSERT)

          { + executeString("create table " + tableName + + " (col1 int4, col2 int4) partition by column(l_shipdate text, l_returnflag text) ").close(); + + assertTrue(catalog.existsTable(DEFAULT_DATABASE_NAME, tableName)); + assertEquals(2, catalog.getTableDesc(DEFAULT_DATABASE_NAME, tableName).getSchema().size()); + assertEquals(4, catalog.getTableDesc(DEFAULT_DATABASE_NAME, tableName).getLogicalSchema().size()); + + executeString( + "insert overwrite into " + tableName + " select l_orderkey, l_partkey, l_shipdate, l_returnflag from lineitem"); + }

          else

          { + executeString( + "create table " + tableName + "(col1 int4, col2 int4) partition by column(l_shipdate text, l_returnflag text) " + + " as select l_orderkey, l_partkey, l_shipdate, l_returnflag from lineitem"); + }

          +
          + assertTrue(client.existTable(tableName));
          +
          + // Like
          + res = executeString("SELECT * FROM " + tableName
          + + " WHERE l_shipdate LIKE '1996%' and l_returnflag = 'N' order by l_shipdate");
          +
          + expectedResult = "col1,col2,l_shipdate,l_returnflag\n" +
          + "-------------------------------\n" +
          + "1,1,1996-03-13,N\n" +
          + "1,1,1996-04-12,N\n";
          +
          + assertEquals(expectedResult, resultSetToString(res));
          + res.close();
          +
          + // Not like
          + res = executeString("SELECT * FROM " + tableName
          + + " WHERE l_shipdate NOT LIKE '1996%' and l_returnflag IN ('R') order by l_shipdate");
          +
          + expectedResult = "col1,col2,l_shipdate,l_returnflag\n" +
          + "-------------------------------\n" +
          + "3,3,1993-11-09,R\n" +
          + "3,2,1994-02-02,R\n";
          +
          + assertEquals(expectedResult, resultSetToString(res));
          + res.close();
          +
          + // In
          + res = executeString("SELECT * FROM " + tableName
          + + " WHERE l_shipdate IN ('1993-11-09', '1994-02-02', '1997-01-28') AND l_returnflag = 'R' order by l_shipdate");
          +
          + expectedResult = "col1,col2,l_shipdate,l_returnflag\n" +
          + "-------------------------------\n" +
          + "3,3,1993-11-09,R\n" +
          + "3,2,1994-02-02,R\n";
          +
          + assertEquals(expectedResult, resultSetToString(res));
          + res.close();
          +
          + // Similar to
          + res = executeString("SELECT * FROM " + tableName + " WHERE l_shipdate similar to '1993%' order by l_shipdate");
          +
          + expectedResult = "col1,col2,l_shipdate,l_returnflag\n" +
          + "-------------------------------\n" +
          + "3,3,1993-11-09,R\n";
          +
          + assertEquals(expectedResult, resultSetToString(res));
          + res.close();
          +
          + // Regular expression
          + res = executeString("SELECT * FROM " + tableName
          + + " WHERE l_shipdate regexp '[1-2][0-9][0-9][3-9][0-1][0-9][0-3][0-9]' "
          + + " AND l_returnflag <> 'N' ORDER BY l_shipdate");
          +
          + expectedResult = "col1,col2,l_shipdate,l_returnflag\n" +
          + "-------------------------------\n" +
          + "3,3,1993-11-09,R\n" +
          + "3,2,1994-02-02,R\n";
          +
          + assertEquals(expectedResult, resultSetToString(res));
          + res.close();
          +
          + // Concatenate
          + res = executeString("SELECT * FROM " + tableName
          + + " WHERE l_shipdate = ( '1996' || '' || '03' || '' || '13' ) order by l_shipdate");
          +
          + expectedResult = "col1,col2,l_shipdate,l_returnflag\n" +
          + "-------------------------------\n" +
          + "1,1,1996-03-13,N\n";
          +
          + assertEquals(expectedResult, resultSetToString(res));
          + res.close();
          +
          + executeString("DROP TABLE " + tableName + " PURGE").close();
          + res.close();
          + }
          +
          + @Test
          + public final void testDatePartitionColumn() throws Exception {
          + ResultSet res = null;
          + String tableName = CatalogUtil.normalizeIdentifier("testDatePartitionColumn");
          + String expectedResult;
          +
          + if (nodeType == NodeType.INSERT)

          { + executeString("create table " + tableName + " (col1 int4, col2 int4) partition by column(key date) ").close(); + + assertTrue(catalog.existsTable(DEFAULT_DATABASE_NAME, tableName)); + assertEquals(2, catalog.getTableDesc(DEFAULT_DATABASE_NAME, tableName).getSchema().size()); + assertEquals(3, catalog.getTableDesc(DEFAULT_DATABASE_NAME, tableName).getLogicalSchema().size()); + + executeString( + "insert overwrite into " + tableName + " select l_orderkey, l_partkey, l_shipdate from lineitem"); + }

          else

          { + executeString( + "create table " + tableName + "(col1 int4, col2 int4) partition by column(key date) " + + " as select l_orderkey, l_partkey, l_shipdate::date from lineitem"); + }

          +
          + assertTrue(client.existTable(tableName));
          +
          + // LessThanOrEquals
          + res = executeString("SELECT * FROM " + tableName + " WHERE key <= date '1995-09-01' order by col1, col2, key");
          +
          + expectedResult = "col1,col2,key\n" +
          + "-------------------------------\n" +
          + "3,2,1994-02-02\n" +
          + "3,3,1993-11-09\n";
          +
          + assertEquals(expectedResult, resultSetToString(res));
          + res.close();
          +
          + // LessThan and GreaterThan
          + res = executeString("SELECT * FROM " + tableName
          + + " WHERE key > to_date('1993-01-01', 'YYYY-MM-DD') " +
          + " and key < to_date('1996-01-01', 'YYYY-MM-DD') order by col1, col2, key desc");
          +
          + expectedResult = "col1,col2,key\n" +
          + "-------------------------------\n" +
          + "3,2,1994-02-02\n" +
          + "3,3,1993-11-09\n";
          +
          + assertEquals(expectedResult, resultSetToString(res));
          + res.close();
          +
          + // Between
          + res = executeString("SELECT * FROM " + tableName
          + + " WHERE key between date '1993-01-01' and date '1997-01-01' order by col1, col2, key desc");
          +
          + expectedResult = "col1,col2,key\n" +
          + "-------------------------------\n" +
          + "1,1,1996-04-12\n" +
          + "1,1,1996-03-13\n" +
          + "3,2,1994-02-02\n" +
          + "3,3,1993-11-09\n";
          +
          + assertEquals(expectedResult, resultSetToString(res));
          + res.close();
          +
          + // Cast
          + res = executeString("SELECT * FROM " + tableName
          + + " WHERE key > '1993-01-01'::date " +
          + " and key < '1997-01-01'::timestamp order by col1, col2, key ");
          +
          + expectedResult = "col1,col2,key\n" +
          + "-------------------------------\n" +
          + "1,1,1996-03-13\n" +
          + "1,1,1996-04-12\n" +
          + "3,2,1994-02-02\n" +
          + "3,3,1993-11-09\n";
          +
          + assertEquals(expectedResult, resultSetToString(res));
          + res.close();
          +
          + // Interval
          + res = executeString("SELECT * FROM " + tableName
          + + " WHERE key > '1993-01-01'::date " +
          + " and key < date '1994-01-01' + interval '1 year' order by col1, col2, key ");
          +
          + expectedResult = "col1,col2,key\n" +
          + "-------------------------------\n" +
          + "3,2,1994-02-02\n" +
          + "3,3,1993-11-09\n";
          +
          + assertEquals(expectedResult, resultSetToString(res));
          + res.close();
          +
          + // DateTime Function #1
          + res = executeString("SELECT * FROM " + tableName
          + + " WHERE key > '1993-01-01'::date " +
          + " and key < add_months(date '1994-01-01', 12) order by col1, col2, key ");
          +
          + assertEquals(expectedResult, resultSetToString(res));
          + res.close();
          +
          + // DateTime Function #2
          + res = executeString("SELECT * FROM " + tableName
          + + " WHERE key > '1993-01-01'::date " +
          + " and key < add_months('1994-01-01'::timestamp, 12) order by col1, col2, key ");
          +
          + assertEquals(expectedResult, resultSetToString(res));
          + res.close();
          +
          + executeString("DROP TABLE " + tableName + " PURGE").close();
          + res.close();
          + }
          +
          + @Test
          + public final void testTimeStampPartitionColumn() throws Exception {
          — End diff –

          Could you rename TimeStamp to Timestamp?

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/624#discussion_r39639478 — Diff: tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestTablePartitions.java — @@ -1325,4 +1332,421 @@ public final void testDuplicatedPartitions() throws Exception { executeString("DROP TABLE " + tableName + " PURGE"); } } + + @Test + public final void testPatternMatchingPredicatesAndStringFunctions() throws Exception { + ResultSet res = null; + String tableName = CatalogUtil.normalizeIdentifier("testPatternMatchingPredicatesAndStringFunctions"); + String expectedResult; + + if (nodeType == NodeType.INSERT) { + executeString("create table " + tableName + + " (col1 int4, col2 int4) partition by column(l_shipdate text, l_returnflag text) ").close(); + + assertTrue(catalog.existsTable(DEFAULT_DATABASE_NAME, tableName)); + assertEquals(2, catalog.getTableDesc(DEFAULT_DATABASE_NAME, tableName).getSchema().size()); + assertEquals(4, catalog.getTableDesc(DEFAULT_DATABASE_NAME, tableName).getLogicalSchema().size()); + + executeString( + "insert overwrite into " + tableName + " select l_orderkey, l_partkey, l_shipdate, l_returnflag from lineitem"); + } else { + executeString( + "create table " + tableName + "(col1 int4, col2 int4) partition by column(l_shipdate text, l_returnflag text) " + + " as select l_orderkey, l_partkey, l_shipdate, l_returnflag from lineitem"); + } + + assertTrue(client.existTable(tableName)); + + // Like + res = executeString("SELECT * FROM " + tableName + + " WHERE l_shipdate LIKE '1996%' and l_returnflag = 'N' order by l_shipdate"); + + expectedResult = "col1,col2,l_shipdate,l_returnflag\n" + + "-------------------------------\n" + + "1,1,1996-03-13,N\n" + + "1,1,1996-04-12,N\n"; + + assertEquals(expectedResult, resultSetToString(res)); + res.close(); + + // Not like + res = executeString("SELECT * FROM " + tableName + + " WHERE l_shipdate NOT LIKE '1996%' and l_returnflag IN ('R') order by l_shipdate"); + + expectedResult = "col1,col2,l_shipdate,l_returnflag\n" + + "-------------------------------\n" + + "3,3,1993-11-09,R\n" + + "3,2,1994-02-02,R\n"; + + assertEquals(expectedResult, resultSetToString(res)); + res.close(); + + // In + res = executeString("SELECT * FROM " + tableName + + " WHERE l_shipdate IN ('1993-11-09', '1994-02-02', '1997-01-28') AND l_returnflag = 'R' order by l_shipdate"); + + expectedResult = "col1,col2,l_shipdate,l_returnflag\n" + + "-------------------------------\n" + + "3,3,1993-11-09,R\n" + + "3,2,1994-02-02,R\n"; + + assertEquals(expectedResult, resultSetToString(res)); + res.close(); + + // Similar to + res = executeString("SELECT * FROM " + tableName + " WHERE l_shipdate similar to '1993%' order by l_shipdate"); + + expectedResult = "col1,col2,l_shipdate,l_returnflag\n" + + "-------------------------------\n" + + "3,3,1993-11-09,R\n"; + + assertEquals(expectedResult, resultSetToString(res)); + res.close(); + + // Regular expression + res = executeString("SELECT * FROM " + tableName + + " WHERE l_shipdate regexp ' [1-2] [0-9] [0-9] [3-9] [0-1] [0-9] [0-3] [0-9] ' " + + " AND l_returnflag <> 'N' ORDER BY l_shipdate"); + + expectedResult = "col1,col2,l_shipdate,l_returnflag\n" + + "-------------------------------\n" + + "3,3,1993-11-09,R\n" + + "3,2,1994-02-02,R\n"; + + assertEquals(expectedResult, resultSetToString(res)); + res.close(); + + // Concatenate + res = executeString("SELECT * FROM " + tableName + + " WHERE l_shipdate = ( '1996' || ' ' || '03' || ' ' || '13' ) order by l_shipdate"); + + expectedResult = "col1,col2,l_shipdate,l_returnflag\n" + + "-------------------------------\n" + + "1,1,1996-03-13,N\n"; + + assertEquals(expectedResult, resultSetToString(res)); + res.close(); + + executeString("DROP TABLE " + tableName + " PURGE").close(); + res.close(); + } + + @Test + public final void testDatePartitionColumn() throws Exception { + ResultSet res = null; + String tableName = CatalogUtil.normalizeIdentifier("testDatePartitionColumn"); + String expectedResult; + + if (nodeType == NodeType.INSERT) { + executeString("create table " + tableName + " (col1 int4, col2 int4) partition by column(key date) ").close(); + + assertTrue(catalog.existsTable(DEFAULT_DATABASE_NAME, tableName)); + assertEquals(2, catalog.getTableDesc(DEFAULT_DATABASE_NAME, tableName).getSchema().size()); + assertEquals(3, catalog.getTableDesc(DEFAULT_DATABASE_NAME, tableName).getLogicalSchema().size()); + + executeString( + "insert overwrite into " + tableName + " select l_orderkey, l_partkey, l_shipdate from lineitem"); + } else { + executeString( + "create table " + tableName + "(col1 int4, col2 int4) partition by column(key date) " + + " as select l_orderkey, l_partkey, l_shipdate::date from lineitem"); + } + + assertTrue(client.existTable(tableName)); + + // LessThanOrEquals + res = executeString("SELECT * FROM " + tableName + " WHERE key <= date '1995-09-01' order by col1, col2, key"); + + expectedResult = "col1,col2,key\n" + + "-------------------------------\n" + + "3,2,1994-02-02\n" + + "3,3,1993-11-09\n"; + + assertEquals(expectedResult, resultSetToString(res)); + res.close(); + + // LessThan and GreaterThan + res = executeString("SELECT * FROM " + tableName + + " WHERE key > to_date('1993-01-01', 'YYYY-MM-DD') " + + " and key < to_date('1996-01-01', 'YYYY-MM-DD') order by col1, col2, key desc"); + + expectedResult = "col1,col2,key\n" + + "-------------------------------\n" + + "3,2,1994-02-02\n" + + "3,3,1993-11-09\n"; + + assertEquals(expectedResult, resultSetToString(res)); + res.close(); + + // Between + res = executeString("SELECT * FROM " + tableName + + " WHERE key between date '1993-01-01' and date '1997-01-01' order by col1, col2, key desc"); + + expectedResult = "col1,col2,key\n" + + "-------------------------------\n" + + "1,1,1996-04-12\n" + + "1,1,1996-03-13\n" + + "3,2,1994-02-02\n" + + "3,3,1993-11-09\n"; + + assertEquals(expectedResult, resultSetToString(res)); + res.close(); + + // Cast + res = executeString("SELECT * FROM " + tableName + + " WHERE key > '1993-01-01'::date " + + " and key < '1997-01-01'::timestamp order by col1, col2, key "); + + expectedResult = "col1,col2,key\n" + + "-------------------------------\n" + + "1,1,1996-03-13\n" + + "1,1,1996-04-12\n" + + "3,2,1994-02-02\n" + + "3,3,1993-11-09\n"; + + assertEquals(expectedResult, resultSetToString(res)); + res.close(); + + // Interval + res = executeString("SELECT * FROM " + tableName + + " WHERE key > '1993-01-01'::date " + + " and key < date '1994-01-01' + interval '1 year' order by col1, col2, key "); + + expectedResult = "col1,col2,key\n" + + "-------------------------------\n" + + "3,2,1994-02-02\n" + + "3,3,1993-11-09\n"; + + assertEquals(expectedResult, resultSetToString(res)); + res.close(); + + // DateTime Function #1 + res = executeString("SELECT * FROM " + tableName + + " WHERE key > '1993-01-01'::date " + + " and key < add_months(date '1994-01-01', 12) order by col1, col2, key "); + + assertEquals(expectedResult, resultSetToString(res)); + res.close(); + + // DateTime Function #2 + res = executeString("SELECT * FROM " + tableName + + " WHERE key > '1993-01-01'::date " + + " and key < add_months('1994-01-01'::timestamp, 12) order by col1, col2, key "); + + assertEquals(expectedResult, resultSetToString(res)); + res.close(); + + executeString("DROP TABLE " + tableName + " PURGE").close(); + res.close(); + } + + @Test + public final void testTimeStampPartitionColumn() throws Exception { — End diff – Could you rename TimeStamp to Timestamp?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on the pull request:

          https://github.com/apache/tajo/pull/624#issuecomment-140765711

          Thank you for your work. The latest patch looks good to me. Here is my +1. I leaved one comment. You can commit it after reflecting my comment if you agree.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on the pull request: https://github.com/apache/tajo/pull/624#issuecomment-140765711 Thank you for your work. The latest patch looks good to me. Here is my +1. I leaved one comment. You can commit it after reflecting my comment if you agree.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user blrunner commented on the pull request:

          https://github.com/apache/tajo/pull/624#issuecomment-140937939

          @hyunsik

          Thank you very much for your kind review. I've just reflected your comment. After finishing the travis CI build, I'll ship it.

          Show
          githubbot ASF GitHub Bot added a comment - Github user blrunner commented on the pull request: https://github.com/apache/tajo/pull/624#issuecomment-140937939 @hyunsik Thank you very much for your kind review. I've just reflected your comment. After finishing the travis CI build, I'll ship it.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/tajo/pull/624

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/tajo/pull/624
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Tajo-master-CODEGEN-build #514 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/514/)
          TAJO-1493: Make partition pruning based on catalog informations. (blrunner: rev b68329101b412649149f261002c53a45f2711d75)

          • tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestAlterTable.java
          • tajo-common/src/main/proto/errors.proto
          • tajo-plan/src/main/java/org/apache/tajo/plan/expr/AlgebraicUtil.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/ColPartitionStoreExec.java
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogService.java
          • tajo-common/src/main/java/org/apache/tajo/exception/PartitionNotFoundException.java
          • tajo-catalog/tajo-catalog-client/src/main/proto/CatalogProtocol.proto
          • tajo-catalog/tajo-catalog-server/pom.xml
          • tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogStore.java
          • tajo-core/src/main/java/org/apache/tajo/master/TajoMasterClientService.java
          • tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto
          • tajo-plan/src/main/java/org/apache/tajo/plan/util/PartitionFilterAlgebraVisitor.java
          • CHANGES
          • tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestTablePartitions.java
          • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/rewrite/rules/PartitionedTableRewriter.java
          • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java
          • tajo-common/src/main/java/org/apache/tajo/exception/ErrorMessages.java
          • tajo-catalog/tajo-catalog-client/src/main/java/org/apache/tajo/catalog/AbstractCatalogClient.java
          • tajo-common/src/main/java/org/apache/tajo/exception/UndefinedPartitionMethodException.java
          • tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalogAgainstCaseSensitivity.java
          • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/CatalogStore.java
          • tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalog.java
          • tajo-catalog/tajo-catalog-drivers/tajo-hive/src/test/java/org/apache/tajo/catalog/store/TestHiveCatalogStore.java
          • tajo-core-tests/src/test/java/org/apache/tajo/engine/planner/TestEvalNodeToExprConverter.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/util/EvalNodeToExprConverter.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Tajo-master-CODEGEN-build #514 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/514/ ) TAJO-1493 : Make partition pruning based on catalog informations. (blrunner: rev b68329101b412649149f261002c53a45f2711d75) tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestAlterTable.java tajo-common/src/main/proto/errors.proto tajo-plan/src/main/java/org/apache/tajo/plan/expr/AlgebraicUtil.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/ColPartitionStoreExec.java tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogService.java tajo-common/src/main/java/org/apache/tajo/exception/PartitionNotFoundException.java tajo-catalog/tajo-catalog-client/src/main/proto/CatalogProtocol.proto tajo-catalog/tajo-catalog-server/pom.xml tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogStore.java tajo-core/src/main/java/org/apache/tajo/master/TajoMasterClientService.java tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto tajo-plan/src/main/java/org/apache/tajo/plan/util/PartitionFilterAlgebraVisitor.java CHANGES tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestTablePartitions.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java tajo-plan/src/main/java/org/apache/tajo/plan/rewrite/rules/PartitionedTableRewriter.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java tajo-common/src/main/java/org/apache/tajo/exception/ErrorMessages.java tajo-catalog/tajo-catalog-client/src/main/java/org/apache/tajo/catalog/AbstractCatalogClient.java tajo-common/src/main/java/org/apache/tajo/exception/UndefinedPartitionMethodException.java tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalogAgainstCaseSensitivity.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/CatalogStore.java tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalog.java tajo-catalog/tajo-catalog-drivers/tajo-hive/src/test/java/org/apache/tajo/catalog/store/TestHiveCatalogStore.java tajo-core-tests/src/test/java/org/apache/tajo/engine/planner/TestEvalNodeToExprConverter.java tajo-plan/src/main/java/org/apache/tajo/plan/util/EvalNodeToExprConverter.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Tajo-master-build #872 (See https://builds.apache.org/job/Tajo-master-build/872/)
          TAJO-1493: Make partition pruning based on catalog informations. (blrunner: rev b68329101b412649149f261002c53a45f2711d75)

          • tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalog.java
          • tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/ColPartitionStoreExec.java
          • tajo-common/src/main/proto/errors.proto
          • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java
          • tajo-catalog/tajo-catalog-client/src/main/java/org/apache/tajo/catalog/AbstractCatalogClient.java
          • tajo-common/src/main/java/org/apache/tajo/exception/UndefinedPartitionMethodException.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/util/PartitionFilterAlgebraVisitor.java
          • tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestTablePartitions.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/util/EvalNodeToExprConverter.java
          • tajo-common/src/main/java/org/apache/tajo/exception/PartitionNotFoundException.java
          • tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogStore.java
          • tajo-catalog/tajo-catalog-client/src/main/proto/CatalogProtocol.proto
          • tajo-core-tests/src/test/java/org/apache/tajo/engine/planner/TestEvalNodeToExprConverter.java
          • tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalogAgainstCaseSensitivity.java
          • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java
          • tajo-catalog/tajo-catalog-server/pom.xml
          • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/CatalogStore.java
          • tajo-catalog/tajo-catalog-drivers/tajo-hive/src/test/java/org/apache/tajo/catalog/store/TestHiveCatalogStore.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/expr/AlgebraicUtil.java
          • tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestAlterTable.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/rewrite/rules/PartitionedTableRewriter.java
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogService.java
          • CHANGES
          • tajo-common/src/main/java/org/apache/tajo/exception/ErrorMessages.java
          • tajo-core/src/main/java/org/apache/tajo/master/TajoMasterClientService.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Tajo-master-build #872 (See https://builds.apache.org/job/Tajo-master-build/872/ ) TAJO-1493 : Make partition pruning based on catalog informations. (blrunner: rev b68329101b412649149f261002c53a45f2711d75) tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalog.java tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/ColPartitionStoreExec.java tajo-common/src/main/proto/errors.proto tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java tajo-catalog/tajo-catalog-client/src/main/java/org/apache/tajo/catalog/AbstractCatalogClient.java tajo-common/src/main/java/org/apache/tajo/exception/UndefinedPartitionMethodException.java tajo-plan/src/main/java/org/apache/tajo/plan/util/PartitionFilterAlgebraVisitor.java tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestTablePartitions.java tajo-plan/src/main/java/org/apache/tajo/plan/util/EvalNodeToExprConverter.java tajo-common/src/main/java/org/apache/tajo/exception/PartitionNotFoundException.java tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogStore.java tajo-catalog/tajo-catalog-client/src/main/proto/CatalogProtocol.proto tajo-core-tests/src/test/java/org/apache/tajo/engine/planner/TestEvalNodeToExprConverter.java tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalogAgainstCaseSensitivity.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java tajo-catalog/tajo-catalog-server/pom.xml tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/CatalogStore.java tajo-catalog/tajo-catalog-drivers/tajo-hive/src/test/java/org/apache/tajo/catalog/store/TestHiveCatalogStore.java tajo-plan/src/main/java/org/apache/tajo/plan/expr/AlgebraicUtil.java tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestAlterTable.java tajo-plan/src/main/java/org/apache/tajo/plan/rewrite/rules/PartitionedTableRewriter.java tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogService.java CHANGES tajo-common/src/main/java/org/apache/tajo/exception/ErrorMessages.java tajo-core/src/main/java/org/apache/tajo/master/TajoMasterClientService.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Tajo-0.11.0-build #49 (See https://builds.apache.org/job/Tajo-0.11.0-build/49/)
          TAJO-1493: Make partition pruning based on catalog informations. (blrunner: rev 84c928dfa135a8108550424b19c69c51683e820d)

          • tajo-catalog/tajo-catalog-client/src/main/proto/CatalogProtocol.proto
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogService.java
          • tajo-catalog/tajo-catalog-drivers/tajo-hive/src/test/java/org/apache/tajo/catalog/store/TestHiveCatalogStore.java
          • tajo-catalog/tajo-catalog-client/src/main/java/org/apache/tajo/catalog/AbstractCatalogClient.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/ColPartitionStoreExec.java
          • tajo-common/src/main/java/org/apache/tajo/exception/PartitionNotFoundException.java
          • tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto
          • tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalogAgainstCaseSensitivity.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/expr/AlgebraicUtil.java
          • tajo-common/src/main/proto/errors.proto
          • tajo-plan/src/main/java/org/apache/tajo/plan/rewrite/rules/PartitionedTableRewriter.java
          • tajo-core/src/main/java/org/apache/tajo/master/TajoMasterClientService.java
          • tajo-core-tests/src/test/java/org/apache/tajo/engine/planner/TestEvalNodeToExprConverter.java
          • tajo-catalog/tajo-catalog-server/pom.xml
          • tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalog.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/util/EvalNodeToExprConverter.java
          • tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogStore.java
          • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/CatalogStore.java
          • tajo-common/src/main/java/org/apache/tajo/exception/ErrorMessages.java
          • tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestTablePartitions.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/util/PartitionFilterAlgebraVisitor.java
          • tajo-common/src/main/java/org/apache/tajo/exception/UndefinedPartitionMethodException.java
          • tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestAlterTable.java
          • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java
          • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java
          • CHANGES
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Tajo-0.11.0-build #49 (See https://builds.apache.org/job/Tajo-0.11.0-build/49/ ) TAJO-1493 : Make partition pruning based on catalog informations. (blrunner: rev 84c928dfa135a8108550424b19c69c51683e820d) tajo-catalog/tajo-catalog-client/src/main/proto/CatalogProtocol.proto tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogService.java tajo-catalog/tajo-catalog-drivers/tajo-hive/src/test/java/org/apache/tajo/catalog/store/TestHiveCatalogStore.java tajo-catalog/tajo-catalog-client/src/main/java/org/apache/tajo/catalog/AbstractCatalogClient.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/ColPartitionStoreExec.java tajo-common/src/main/java/org/apache/tajo/exception/PartitionNotFoundException.java tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalogAgainstCaseSensitivity.java tajo-plan/src/main/java/org/apache/tajo/plan/expr/AlgebraicUtil.java tajo-common/src/main/proto/errors.proto tajo-plan/src/main/java/org/apache/tajo/plan/rewrite/rules/PartitionedTableRewriter.java tajo-core/src/main/java/org/apache/tajo/master/TajoMasterClientService.java tajo-core-tests/src/test/java/org/apache/tajo/engine/planner/TestEvalNodeToExprConverter.java tajo-catalog/tajo-catalog-server/pom.xml tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalog.java tajo-plan/src/main/java/org/apache/tajo/plan/util/EvalNodeToExprConverter.java tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogStore.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/CatalogStore.java tajo-common/src/main/java/org/apache/tajo/exception/ErrorMessages.java tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestTablePartitions.java tajo-plan/src/main/java/org/apache/tajo/plan/util/PartitionFilterAlgebraVisitor.java tajo-common/src/main/java/org/apache/tajo/exception/UndefinedPartitionMethodException.java tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestAlterTable.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java CHANGES
          Hide
          blrunner Jaehwa Jung added a comment -

          I'll reopen this issue because it includes some critical bugs.

          Show
          blrunner Jaehwa Jung added a comment - I'll reopen this issue because it includes some critical bugs.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Tajo-master-CODEGEN-build #517 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/517/)
          Revert "TAJO-1493: Make partition pruning based on catalog informations." (blrunner: rev e1c2d352e15f67d148700fe61d78a37640341dab)

          • tajo-catalog/tajo-catalog-client/src/main/proto/CatalogProtocol.proto
          • tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestAlterTable.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/util/PartitionFilterAlgebraVisitor.java
          • tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalogAgainstCaseSensitivity.java
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogService.java
          • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/CatalogStore.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/ColPartitionStoreExec.java
          • tajo-common/src/main/proto/errors.proto
          • tajo-plan/src/main/java/org/apache/tajo/plan/rewrite/rules/PartitionedTableRewriter.java
          • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java
          • tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestTablePartitions.java
          • tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalog.java
          • tajo-core-tests/src/test/java/org/apache/tajo/engine/planner/TestEvalNodeToExprConverter.java
          • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java
          • tajo-common/src/main/java/org/apache/tajo/exception/PartitionNotFoundException.java
          • tajo-catalog/tajo-catalog-client/src/main/java/org/apache/tajo/catalog/AbstractCatalogClient.java
          • tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto
          • tajo-catalog/tajo-catalog-drivers/tajo-hive/src/test/java/org/apache/tajo/catalog/store/TestHiveCatalogStore.java
          • tajo-core/src/main/java/org/apache/tajo/master/TajoMasterClientService.java
          • tajo-common/src/main/java/org/apache/tajo/exception/ErrorMessages.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/expr/AlgebraicUtil.java
          • tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogStore.java
          • tajo-common/src/main/java/org/apache/tajo/exception/UndefinedPartitionMethodException.java
          • CHANGES
          • tajo-catalog/tajo-catalog-server/pom.xml
          • tajo-plan/src/main/java/org/apache/tajo/plan/util/EvalNodeToExprConverter.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Tajo-master-CODEGEN-build #517 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/517/ ) Revert " TAJO-1493 : Make partition pruning based on catalog informations." (blrunner: rev e1c2d352e15f67d148700fe61d78a37640341dab) tajo-catalog/tajo-catalog-client/src/main/proto/CatalogProtocol.proto tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestAlterTable.java tajo-plan/src/main/java/org/apache/tajo/plan/util/PartitionFilterAlgebraVisitor.java tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalogAgainstCaseSensitivity.java tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogService.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/CatalogStore.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/ColPartitionStoreExec.java tajo-common/src/main/proto/errors.proto tajo-plan/src/main/java/org/apache/tajo/plan/rewrite/rules/PartitionedTableRewriter.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestTablePartitions.java tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalog.java tajo-core-tests/src/test/java/org/apache/tajo/engine/planner/TestEvalNodeToExprConverter.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java tajo-common/src/main/java/org/apache/tajo/exception/PartitionNotFoundException.java tajo-catalog/tajo-catalog-client/src/main/java/org/apache/tajo/catalog/AbstractCatalogClient.java tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto tajo-catalog/tajo-catalog-drivers/tajo-hive/src/test/java/org/apache/tajo/catalog/store/TestHiveCatalogStore.java tajo-core/src/main/java/org/apache/tajo/master/TajoMasterClientService.java tajo-common/src/main/java/org/apache/tajo/exception/ErrorMessages.java tajo-plan/src/main/java/org/apache/tajo/plan/expr/AlgebraicUtil.java tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogStore.java tajo-common/src/main/java/org/apache/tajo/exception/UndefinedPartitionMethodException.java CHANGES tajo-catalog/tajo-catalog-server/pom.xml tajo-plan/src/main/java/org/apache/tajo/plan/util/EvalNodeToExprConverter.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Tajo-master-build #875 (See https://builds.apache.org/job/Tajo-master-build/875/)
          Revert "TAJO-1493: Make partition pruning based on catalog informations." (blrunner: rev e1c2d352e15f67d148700fe61d78a37640341dab)

          • tajo-common/src/main/java/org/apache/tajo/exception/UndefinedPartitionMethodException.java
          • tajo-common/src/main/java/org/apache/tajo/exception/ErrorMessages.java
          • tajo-catalog/tajo-catalog-client/src/main/proto/CatalogProtocol.proto
          • tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalogAgainstCaseSensitivity.java
          • CHANGES
          • tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogStore.java
          • tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestAlterTable.java
          • tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalog.java
          • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/rewrite/rules/PartitionedTableRewriter.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/util/EvalNodeToExprConverter.java
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogService.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/expr/AlgebraicUtil.java
          • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/CatalogStore.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/util/PartitionFilterAlgebraVisitor.java
          • tajo-catalog/tajo-catalog-server/pom.xml
          • tajo-catalog/tajo-catalog-drivers/tajo-hive/src/test/java/org/apache/tajo/catalog/store/TestHiveCatalogStore.java
          • tajo-catalog/tajo-catalog-client/src/main/java/org/apache/tajo/catalog/AbstractCatalogClient.java
          • tajo-common/src/main/java/org/apache/tajo/exception/PartitionNotFoundException.java
          • tajo-common/src/main/proto/errors.proto
          • tajo-core/src/main/java/org/apache/tajo/master/TajoMasterClientService.java
          • tajo-core-tests/src/test/java/org/apache/tajo/engine/planner/TestEvalNodeToExprConverter.java
          • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java
          • tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto
          • tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestTablePartitions.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/ColPartitionStoreExec.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Tajo-master-build #875 (See https://builds.apache.org/job/Tajo-master-build/875/ ) Revert " TAJO-1493 : Make partition pruning based on catalog informations." (blrunner: rev e1c2d352e15f67d148700fe61d78a37640341dab) tajo-common/src/main/java/org/apache/tajo/exception/UndefinedPartitionMethodException.java tajo-common/src/main/java/org/apache/tajo/exception/ErrorMessages.java tajo-catalog/tajo-catalog-client/src/main/proto/CatalogProtocol.proto tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalogAgainstCaseSensitivity.java CHANGES tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogStore.java tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestAlterTable.java tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalog.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java tajo-plan/src/main/java/org/apache/tajo/plan/rewrite/rules/PartitionedTableRewriter.java tajo-plan/src/main/java/org/apache/tajo/plan/util/EvalNodeToExprConverter.java tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogService.java tajo-plan/src/main/java/org/apache/tajo/plan/expr/AlgebraicUtil.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/CatalogStore.java tajo-plan/src/main/java/org/apache/tajo/plan/util/PartitionFilterAlgebraVisitor.java tajo-catalog/tajo-catalog-server/pom.xml tajo-catalog/tajo-catalog-drivers/tajo-hive/src/test/java/org/apache/tajo/catalog/store/TestHiveCatalogStore.java tajo-catalog/tajo-catalog-client/src/main/java/org/apache/tajo/catalog/AbstractCatalogClient.java tajo-common/src/main/java/org/apache/tajo/exception/PartitionNotFoundException.java tajo-common/src/main/proto/errors.proto tajo-core/src/main/java/org/apache/tajo/master/TajoMasterClientService.java tajo-core-tests/src/test/java/org/apache/tajo/engine/planner/TestEvalNodeToExprConverter.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestTablePartitions.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/ColPartitionStoreExec.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Tajo-0.11.0-build #52 (See https://builds.apache.org/job/Tajo-0.11.0-build/52/)
          Revert "TAJO-1493: Make partition pruning based on catalog informations." (blrunner: rev 35dc8df4c7a19f4f0bc04cffe5d6153be588d830)

          • tajo-catalog/tajo-catalog-client/src/main/proto/CatalogProtocol.proto
          • tajo-catalog/tajo-catalog-server/pom.xml
          • tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto
          • tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalogAgainstCaseSensitivity.java
          • tajo-common/src/main/proto/errors.proto
          • CHANGES
          • tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalog.java
          • tajo-catalog/tajo-catalog-drivers/tajo-hive/src/test/java/org/apache/tajo/catalog/store/TestHiveCatalogStore.java
          • tajo-common/src/main/java/org/apache/tajo/exception/PartitionNotFoundException.java
          • tajo-core-tests/src/test/java/org/apache/tajo/engine/planner/TestEvalNodeToExprConverter.java
          • tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogStore.java
          • tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestTablePartitions.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/ColPartitionStoreExec.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/expr/AlgebraicUtil.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/rewrite/rules/PartitionedTableRewriter.java
          • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/util/EvalNodeToExprConverter.java
          • tajo-common/src/main/java/org/apache/tajo/exception/UndefinedPartitionMethodException.java
          • tajo-common/src/main/java/org/apache/tajo/exception/ErrorMessages.java
          • tajo-core/src/main/java/org/apache/tajo/master/TajoMasterClientService.java
          • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/CatalogStore.java
          • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java
          • tajo-catalog/tajo-catalog-client/src/main/java/org/apache/tajo/catalog/AbstractCatalogClient.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/util/PartitionFilterAlgebraVisitor.java
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogService.java
          • tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestAlterTable.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Tajo-0.11.0-build #52 (See https://builds.apache.org/job/Tajo-0.11.0-build/52/ ) Revert " TAJO-1493 : Make partition pruning based on catalog informations." (blrunner: rev 35dc8df4c7a19f4f0bc04cffe5d6153be588d830) tajo-catalog/tajo-catalog-client/src/main/proto/CatalogProtocol.proto tajo-catalog/tajo-catalog-server/pom.xml tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalogAgainstCaseSensitivity.java tajo-common/src/main/proto/errors.proto CHANGES tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalog.java tajo-catalog/tajo-catalog-drivers/tajo-hive/src/test/java/org/apache/tajo/catalog/store/TestHiveCatalogStore.java tajo-common/src/main/java/org/apache/tajo/exception/PartitionNotFoundException.java tajo-core-tests/src/test/java/org/apache/tajo/engine/planner/TestEvalNodeToExprConverter.java tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogStore.java tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestTablePartitions.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/ColPartitionStoreExec.java tajo-plan/src/main/java/org/apache/tajo/plan/expr/AlgebraicUtil.java tajo-plan/src/main/java/org/apache/tajo/plan/rewrite/rules/PartitionedTableRewriter.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java tajo-plan/src/main/java/org/apache/tajo/plan/util/EvalNodeToExprConverter.java tajo-common/src/main/java/org/apache/tajo/exception/UndefinedPartitionMethodException.java tajo-common/src/main/java/org/apache/tajo/exception/ErrorMessages.java tajo-core/src/main/java/org/apache/tajo/master/TajoMasterClientService.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/CatalogStore.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java tajo-catalog/tajo-catalog-client/src/main/java/org/apache/tajo/catalog/AbstractCatalogClient.java tajo-plan/src/main/java/org/apache/tajo/plan/util/PartitionFilterAlgebraVisitor.java tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogService.java tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestAlterTable.java
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user blrunner opened a pull request:

          https://github.com/apache/tajo/pull/772

          TAJO-1493: Make partition pruning based on catalog informations.

          Reopen PR which contains following features.

          • Add columns for partition volume and file count to Catalog
          • Allow removed partition directories in partitioned table.
          • Implement PartitionNotFoundException handling
          • Implement UnsupportedException handling
          • Add unit test cases for abnormal partition directories

          *NOTE: old PR is https://github.com/apache/tajo/pull/624.*

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/blrunner/tajo TAJO-1493

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/tajo/pull/772.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #772


          commit 743122bade809e5a6fd6ccbb5025a87233e0ac40
          Author: JaeHwa Jung <blrunner@apache.org>
          Date: 2015-07-06T01:49:20Z

          TAJO-1493: Add a method to get partition directories with filter conditions.

          commit 422dbb10b53d3f97f97d9723248e905e3cc76e78
          Author: JaeHwa Jung <blrunner@apache.org>
          Date: 2015-07-06T06:28:26Z

          Fix updateTableStats error

          commit 515f0e364c31150e518d0a139e3bb4728fbf36ca
          Author: JaeHwa Jung <blrunner@apache.org>
          Date: 2015-07-10T02:49:27Z

          Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1493

          commit 23297030c6d4d033f57cac39d2b710de590306bd
          Author: JaeHwa Jung <blrunner@apache.org>
          Date: 2015-07-14T07:18:18Z

          Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1493

          commit 7cb8fa81302d070e3d1c5c0e87f3d39ba574c5ce
          Author: JaeHwa Jung <blrunner@apache.org>
          Date: 2015-07-17T10:28:56Z

          Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1493

          Conflicts:
          tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalog.java

          commit 2073d04dbab81449e0ee27222c9897d9a07ada7c
          Author: JaeHwa Jung <blrunner@apache.org>
          Date: 2015-07-24T02:54:47Z

          Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1493

          Conflicts:
          tajo-catalog/tajo-catalog-client/src/main/java/org/apache/tajo/catalog/AbstractCatalogClient.java
          tajo-catalog/tajo-catalog-client/src/main/proto/CatalogProtocol.proto
          tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogService.java
          tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java
          tajo-plan/src/main/java/org/apache/tajo/plan/rewrite/rules/PartitionedTableRewriter.java

          commit 1c615b598d1acb9e186170b8eca5f9ea8c657321
          Author: JaeHwa Jung <blrunner@apache.org>
          Date: 2015-07-31T03:14:40Z

          Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1493

          Conflicts:
          tajo-catalog/tajo-catalog-client/src/main/proto/CatalogProtocol.proto
          tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java

          commit a7f7035583bd16454d6596af03adce390f5d9ec7
          Author: JaeHwa Jung <blrunner@apache.org>
          Date: 2015-08-02T16:19:35Z

          Implement getPartitionsByDirectSql and fix some bugs

          commit ff19c4934b8032158a7a8d02756f2688c61b8043
          Author: JaeHwa Jung <blrunner@apache.org>
          Date: 2015-08-02T16:33:09Z

          Remove unnecessary updates

          commit d30a6eb66a964aaab0318abe0b342a80737a7d36
          Author: JaeHwa Jung <blrunner@apache.org>
          Date: 2015-08-02T16:34:25Z

          Remove unnecessary codes

          commit e492d68225e21023811aa308f2ac56a81218ff50
          Author: JaeHwa Jung <blrunner@apache.org>
          Date: 2015-08-03T07:13:55Z

          Optimize direct sql and add more description.

          commit f40e36d4aaa801aa0b660f30ba25ead92bb9a2d2
          Author: JaeHwa Jung <blrunner@apache.org>
          Date: 2015-08-03T07:32:27Z

          Rename SQLFinderWithPartitionFilter to PartitionDirectSQLBuilder

          commit 387ba94f5e5eea179bf4355413bee0302fd21706
          Author: JaeHwa Jung <blrunner@apache.org>
          Date: 2015-08-03T09:35:04Z

          Implement HiveCatalogStore::getPartitionsByDirectSql

          commit c470489646e288d1e7a94e17cc066a559bcc1e29
          Author: JaeHwa Jung <blrunner@apache.org>
          Date: 2015-08-03T13:32:20Z

          Fix errors for addPartitions and getPartitionsByDirectSql

          commit 560c202495651148076e4f44541de321a21fb021
          Author: JaeHwa Jung <blrunner@apache.org>
          Date: 2015-08-03T15:08:29Z

          Add unit test cases for PartitionedTableRewriter

          commit 15c5082ba2433ea487dae11ead9f2f94fb4aeafb
          Author: JaeHwa Jung <blrunner@apache.org>
          Date: 2015-08-03T15:22:50Z

          Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1493

          commit 28cbde04bb3a5144038430f333745cc32800c0f4
          Author: JaeHwa Jung <blrunner@apache.org>
          Date: 2015-08-04T05:12:14Z

          Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1493

          Conflicts:
          tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogService.java
          tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/MemStore.java

          commit b9eadd2f627dc55595192f218daba0b58db59f80
          Author: JaeHwa Jung <blrunner@apache.org>
          Date: 2015-08-06T17:35:18Z

          Remove unnecessary codes

          commit f1a0fab0cbd01090e6d26b4167ecba0a750d9303
          Author: JaeHwa Jung <blrunner@apache.org>
          Date: 2015-08-06T18:01:46Z

          Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1493

          Conflicts:
          tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java

          commit 7f53fc5de593beee8dbc5f80f5fcb20ce1dca00b
          Author: JaeHwa Jung <blrunner@apache.org>
          Date: 2015-08-06T18:08:53Z

          Remove unused packages.

          commit 4aa8ba14ecd1fc4eebec3b6158b2857d1e83d23f
          Author: JaeHwa Jung <blrunner@apache.org>
          Date: 2015-08-20T08:38:47Z

          Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1493

          Conflicts:
          tajo-catalog/tajo-catalog-client/src/main/java/org/apache/tajo/catalog/AbstractCatalogClient.java
          tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogService.java
          tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogStore.java
          tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java
          tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/CatalogStore.java
          tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/MemStore.java
          tajo-common/src/main/java/org/apache/tajo/exception/ErrorMessages.java

          commit 03ea651b45f8ae025243b20f6df86393db49cc15
          Author: JaeHwa Jung <blrunner@apache.org>
          Date: 2015-08-24T03:11:35Z

          Add ScanQualConverter

          commit 5a405373917e337a983bf6c4fbed9340840197d9
          Author: JaeHwa Jung <blrunner@apache.org>
          Date: 2015-08-24T03:12:03Z

          Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1493

          commit 43c8be02bbdb5c52e3cb292974bb15eab09be42e
          Author: JaeHwa Jung <blrunner@apache.org>
          Date: 2015-08-24T09:12:45Z

          Implement CatalogStore::getPartitionsByAlgebra

          commit 63a02091ce8506e24d269d9869e5f253efe7f409
          Author: JaeHwa Jung <blrunner@apache.org>
          Date: 2015-08-24T09:19:16Z

          Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1493

          commit 0db124cdb28faa57cab005a5c9d481202b9c1a8c
          Author: JaeHwa Jung <blrunner@apache.org>
          Date: 2015-08-24T17:07:44Z

          Fix unit test errors and remove unnecessary codes

          commit 7e04346dfaacd4dedfe84716baeb13e6e4e4cf5e
          Author: JaeHwa Jung <blrunner@apache.org>
          Date: 2015-08-24T23:56:34Z

          Rename method name and unused codes

          commit 9aeb2f792f5cd7c58202e475a8728cf30a6f6c9f
          Author: JaeHwa Jung <blrunner@apache.org>
          Date: 2015-08-25T01:14:30Z

          Add UNDEFINED_PARTITIONS

          commit 5000c3bc6c8aa29b526d25ce0423f79823515874
          Author: JaeHwa Jung <blrunner@apache.org>
          Date: 2015-08-25T01:59:57Z

          Fix mismatched argument bug

          commit 8982438882c3506a2c307bf650c8ed436346c33e
          Author: JaeHwa Jung <blrunner@apache.org>
          Date: 2015-08-31T01:50:24Z

          Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1493


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user blrunner opened a pull request: https://github.com/apache/tajo/pull/772 TAJO-1493 : Make partition pruning based on catalog informations. Reopen PR which contains following features. Add columns for partition volume and file count to Catalog Allow removed partition directories in partitioned table. Implement PartitionNotFoundException handling Implement UnsupportedException handling Add unit test cases for abnormal partition directories *NOTE: old PR is https://github.com/apache/tajo/pull/624.* You can merge this pull request into a Git repository by running: $ git pull https://github.com/blrunner/tajo TAJO-1493 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/tajo/pull/772.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #772 commit 743122bade809e5a6fd6ccbb5025a87233e0ac40 Author: JaeHwa Jung <blrunner@apache.org> Date: 2015-07-06T01:49:20Z TAJO-1493 : Add a method to get partition directories with filter conditions. commit 422dbb10b53d3f97f97d9723248e905e3cc76e78 Author: JaeHwa Jung <blrunner@apache.org> Date: 2015-07-06T06:28:26Z Fix updateTableStats error commit 515f0e364c31150e518d0a139e3bb4728fbf36ca Author: JaeHwa Jung <blrunner@apache.org> Date: 2015-07-10T02:49:27Z Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1493 commit 23297030c6d4d033f57cac39d2b710de590306bd Author: JaeHwa Jung <blrunner@apache.org> Date: 2015-07-14T07:18:18Z Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1493 commit 7cb8fa81302d070e3d1c5c0e87f3d39ba574c5ce Author: JaeHwa Jung <blrunner@apache.org> Date: 2015-07-17T10:28:56Z Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1493 Conflicts: tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalog.java commit 2073d04dbab81449e0ee27222c9897d9a07ada7c Author: JaeHwa Jung <blrunner@apache.org> Date: 2015-07-24T02:54:47Z Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1493 Conflicts: tajo-catalog/tajo-catalog-client/src/main/java/org/apache/tajo/catalog/AbstractCatalogClient.java tajo-catalog/tajo-catalog-client/src/main/proto/CatalogProtocol.proto tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogService.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java tajo-plan/src/main/java/org/apache/tajo/plan/rewrite/rules/PartitionedTableRewriter.java commit 1c615b598d1acb9e186170b8eca5f9ea8c657321 Author: JaeHwa Jung <blrunner@apache.org> Date: 2015-07-31T03:14:40Z Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1493 Conflicts: tajo-catalog/tajo-catalog-client/src/main/proto/CatalogProtocol.proto tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java commit a7f7035583bd16454d6596af03adce390f5d9ec7 Author: JaeHwa Jung <blrunner@apache.org> Date: 2015-08-02T16:19:35Z Implement getPartitionsByDirectSql and fix some bugs commit ff19c4934b8032158a7a8d02756f2688c61b8043 Author: JaeHwa Jung <blrunner@apache.org> Date: 2015-08-02T16:33:09Z Remove unnecessary updates commit d30a6eb66a964aaab0318abe0b342a80737a7d36 Author: JaeHwa Jung <blrunner@apache.org> Date: 2015-08-02T16:34:25Z Remove unnecessary codes commit e492d68225e21023811aa308f2ac56a81218ff50 Author: JaeHwa Jung <blrunner@apache.org> Date: 2015-08-03T07:13:55Z Optimize direct sql and add more description. commit f40e36d4aaa801aa0b660f30ba25ead92bb9a2d2 Author: JaeHwa Jung <blrunner@apache.org> Date: 2015-08-03T07:32:27Z Rename SQLFinderWithPartitionFilter to PartitionDirectSQLBuilder commit 387ba94f5e5eea179bf4355413bee0302fd21706 Author: JaeHwa Jung <blrunner@apache.org> Date: 2015-08-03T09:35:04Z Implement HiveCatalogStore::getPartitionsByDirectSql commit c470489646e288d1e7a94e17cc066a559bcc1e29 Author: JaeHwa Jung <blrunner@apache.org> Date: 2015-08-03T13:32:20Z Fix errors for addPartitions and getPartitionsByDirectSql commit 560c202495651148076e4f44541de321a21fb021 Author: JaeHwa Jung <blrunner@apache.org> Date: 2015-08-03T15:08:29Z Add unit test cases for PartitionedTableRewriter commit 15c5082ba2433ea487dae11ead9f2f94fb4aeafb Author: JaeHwa Jung <blrunner@apache.org> Date: 2015-08-03T15:22:50Z Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1493 commit 28cbde04bb3a5144038430f333745cc32800c0f4 Author: JaeHwa Jung <blrunner@apache.org> Date: 2015-08-04T05:12:14Z Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1493 Conflicts: tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogService.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/MemStore.java commit b9eadd2f627dc55595192f218daba0b58db59f80 Author: JaeHwa Jung <blrunner@apache.org> Date: 2015-08-06T17:35:18Z Remove unnecessary codes commit f1a0fab0cbd01090e6d26b4167ecba0a750d9303 Author: JaeHwa Jung <blrunner@apache.org> Date: 2015-08-06T18:01:46Z Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1493 Conflicts: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java commit 7f53fc5de593beee8dbc5f80f5fcb20ce1dca00b Author: JaeHwa Jung <blrunner@apache.org> Date: 2015-08-06T18:08:53Z Remove unused packages. commit 4aa8ba14ecd1fc4eebec3b6158b2857d1e83d23f Author: JaeHwa Jung <blrunner@apache.org> Date: 2015-08-20T08:38:47Z Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1493 Conflicts: tajo-catalog/tajo-catalog-client/src/main/java/org/apache/tajo/catalog/AbstractCatalogClient.java tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogService.java tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogStore.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/CatalogStore.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/MemStore.java tajo-common/src/main/java/org/apache/tajo/exception/ErrorMessages.java commit 03ea651b45f8ae025243b20f6df86393db49cc15 Author: JaeHwa Jung <blrunner@apache.org> Date: 2015-08-24T03:11:35Z Add ScanQualConverter commit 5a405373917e337a983bf6c4fbed9340840197d9 Author: JaeHwa Jung <blrunner@apache.org> Date: 2015-08-24T03:12:03Z Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1493 commit 43c8be02bbdb5c52e3cb292974bb15eab09be42e Author: JaeHwa Jung <blrunner@apache.org> Date: 2015-08-24T09:12:45Z Implement CatalogStore::getPartitionsByAlgebra commit 63a02091ce8506e24d269d9869e5f253efe7f409 Author: JaeHwa Jung <blrunner@apache.org> Date: 2015-08-24T09:19:16Z Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1493 commit 0db124cdb28faa57cab005a5c9d481202b9c1a8c Author: JaeHwa Jung <blrunner@apache.org> Date: 2015-08-24T17:07:44Z Fix unit test errors and remove unnecessary codes commit 7e04346dfaacd4dedfe84716baeb13e6e4e4cf5e Author: JaeHwa Jung <blrunner@apache.org> Date: 2015-08-24T23:56:34Z Rename method name and unused codes commit 9aeb2f792f5cd7c58202e475a8728cf30a6f6c9f Author: JaeHwa Jung <blrunner@apache.org> Date: 2015-08-25T01:14:30Z Add UNDEFINED_PARTITIONS commit 5000c3bc6c8aa29b526d25ce0423f79823515874 Author: JaeHwa Jung <blrunner@apache.org> Date: 2015-08-25T01:59:57Z Fix mismatched argument bug commit 8982438882c3506a2c307bf650c8ed436346c33e Author: JaeHwa Jung <blrunner@apache.org> Date: 2015-08-31T01:50:24Z Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1493
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on the pull request:

          https://github.com/apache/tajo/pull/772#issuecomment-141881828

          I'm reviewing.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on the pull request: https://github.com/apache/tajo/pull/772#issuecomment-141881828 I'm reviewing.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r39942689

          — Diff: tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java —
          @@ -812,6 +812,22 @@ public static AlterTableDesc setProperty(String tableName, KeyValueSet params, A
          /**

          • Converts passed parameters to a AlterTableDesc. This method would be called when adding a partition or dropping
          • a table. This creates AlterTableDesc that is a wrapper class for protocol buffer.
            + * *
            + * @param tableName
            + * @param columns
            + * @param values
            + * @param location
            + * @param alterTableType
            + * @return
            + */
            + public static AlterTableDesc addOrDropPartition(String tableName, String[] columns,
            + String[] values, String location, AlterTableType alterTableType) {
              • End diff –

          ```location``` looks confused to me. How about using 'path'?

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r39942689 — Diff: tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java — @@ -812,6 +812,22 @@ public static AlterTableDesc setProperty(String tableName, KeyValueSet params, A /** Converts passed parameters to a AlterTableDesc. This method would be called when adding a partition or dropping a table. This creates AlterTableDesc that is a wrapper class for protocol buffer. + * * + * @param tableName + * @param columns + * @param values + * @param location + * @param alterTableType + * @return + */ + public static AlterTableDesc addOrDropPartition(String tableName, String[] columns, + String[] values, String location, AlterTableType alterTableType) { End diff – ```location``` looks confused to me. How about using 'path'?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r39942690

          — Diff: tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java —
          @@ -821,7 +837,8 @@ public static AlterTableDesc setProperty(String tableName, KeyValueSet params, A

          • @return AlterTableDesc
            */
            public static AlterTableDesc addOrDropPartition(String tableName, String[] columns,
          • String[] values, String location, AlterTableType alterTableType) {
            + String[] values, String location, AlterTableType alterTableType,
              • End diff –

          ```location``` looks confused to me. How about using 'path'?

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r39942690 — Diff: tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java — @@ -821,7 +837,8 @@ public static AlterTableDesc setProperty(String tableName, KeyValueSet params, A @return AlterTableDesc */ public static AlterTableDesc addOrDropPartition(String tableName, String[] columns, String[] values, String location, AlterTableType alterTableType) { + String[] values, String location, AlterTableType alterTableType, End diff – ```location``` looks confused to me. How about using 'path'?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r39942934

          — Diff: tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto —
          @@ -262,6 +263,18 @@ message PartitionIdentifierProto

          { optional string partitionName = 3; }

          +message PartitionsByAlgebraProto {
          + required string databaseName = 1;
          + required string tableName = 2;
          + required string algebra = 3;
          — End diff –

          It would be better if you leave a comment of how the algebra is serialized into a string.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r39942934 — Diff: tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto — @@ -262,6 +263,18 @@ message PartitionIdentifierProto { optional string partitionName = 3; } +message PartitionsByAlgebraProto { + required string databaseName = 1; + required string tableName = 2; + required string algebra = 3; — End diff – It would be better if you leave a comment of how the algebra is serialized into a string.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r39942937

          — Diff: tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto —
          @@ -262,6 +263,18 @@ message PartitionIdentifierProto

          { optional string partitionName = 3; }

          +message PartitionsByAlgebraProto

          { + required string databaseName = 1; + required string tableName = 2; + required string algebra = 3; +}

          +
          +message PartitionsByFilterProto {
          + required string databaseName = 1;
          + required string tableName = 2;
          + required string filter = 3;
          — End diff –

          It would be better if you leave a comment of how the filter is serialized into a string.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r39942937 — Diff: tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto — @@ -262,6 +263,18 @@ message PartitionIdentifierProto { optional string partitionName = 3; } +message PartitionsByAlgebraProto { + required string databaseName = 1; + required string tableName = 2; + required string algebra = 3; +} + +message PartitionsByFilterProto { + required string databaseName = 1; + required string tableName = 2; + required string filter = 3; — End diff – It would be better if you leave a comment of how the filter is serialized into a string.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r39943150

          — Diff: tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogStore.java —
          @@ -845,11 +856,155 @@ public boolean existPartitionMethod(String databaseName, String tableName) throw
          }

          @Override

          • public List<CatalogProtos.PartitionDescProto> getPartitions(String databaseName,
          • String tableName) {
          • throw new UnsupportedOperationException();
            + public List<CatalogProtos.PartitionDescProto> getAllPartitions(String databaseName, String tableName)
            + throws UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException,
            + PartitionNotFoundException
            Unknown macro: { + PartitionsByFilterProto.Builder request = PartitionsByFilterProto.newBuilder(); + request.setDatabaseName(databaseName); + request.setTableName(tableName); + request.setFilter(""); + + List<PartitionDescProto> partitions = getPartitionsByFilter(request.build()); + if (partitions.size() == 0) { + throw new PartitionNotFoundException(tableName); + } else { + return partitions; + } + }

            +
            + @Override
            + public List<PartitionDescProto> getPartitionsByAlgebra(PartitionsByAlgebraProto request) throws
            + UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException,
            + PartitionNotFoundException, UnsupportedException {
            +
            + List<PartitionDescProto> list = null;
            +
            + try

            { + String databaseName = request.getDatabaseName(); + String tableName = request.getTableName(); + + TableDescProto tableDesc = getTable(databaseName, tableName); + String filter = getFilter(databaseName, tableName, tableDesc.getPartition().getExpressionSchema().getFieldsList() + , request.getAlgebra()); + list = getPartitionsByFilterFromHiveMetaStore(databaseName, tableName, filter); + }

            catch (UnsupportedException ue) {

              • End diff –

          Who does throw this exception?

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r39943150 — Diff: tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogStore.java — @@ -845,11 +856,155 @@ public boolean existPartitionMethod(String databaseName, String tableName) throw } @Override public List<CatalogProtos.PartitionDescProto> getPartitions(String databaseName, String tableName) { throw new UnsupportedOperationException(); + public List<CatalogProtos.PartitionDescProto> getAllPartitions(String databaseName, String tableName) + throws UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException, + PartitionNotFoundException Unknown macro: { + PartitionsByFilterProto.Builder request = PartitionsByFilterProto.newBuilder(); + request.setDatabaseName(databaseName); + request.setTableName(tableName); + request.setFilter(""); + + List<PartitionDescProto> partitions = getPartitionsByFilter(request.build()); + if (partitions.size() == 0) { + throw new PartitionNotFoundException(tableName); + } else { + return partitions; + } + } + + @Override + public List<PartitionDescProto> getPartitionsByAlgebra(PartitionsByAlgebraProto request) throws + UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException, + PartitionNotFoundException, UnsupportedException { + + List<PartitionDescProto> list = null; + + try { + String databaseName = request.getDatabaseName(); + String tableName = request.getTableName(); + + TableDescProto tableDesc = getTable(databaseName, tableName); + String filter = getFilter(databaseName, tableName, tableDesc.getPartition().getExpressionSchema().getFieldsList() + , request.getAlgebra()); + list = getPartitionsByFilterFromHiveMetaStore(databaseName, tableName, filter); + } catch (UnsupportedException ue) { End diff – Who does throw this exception?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r39943564

          — Diff: tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogStore.java —
          @@ -845,11 +856,155 @@ public boolean existPartitionMethod(String databaseName, String tableName) throw
          }

          @Override

          • public List<CatalogProtos.PartitionDescProto> getPartitions(String databaseName,
          • String tableName) {
          • throw new UnsupportedOperationException();
            + public List<CatalogProtos.PartitionDescProto> getAllPartitions(String databaseName, String tableName)
            + throws UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException,
            + PartitionNotFoundException
            Unknown macro: { + PartitionsByFilterProto.Builder request = PartitionsByFilterProto.newBuilder(); + request.setDatabaseName(databaseName); + request.setTableName(tableName); + request.setFilter(""); + + List<PartitionDescProto> partitions = getPartitionsByFilter(request.build()); + if (partitions.size() == 0) { + throw new PartitionNotFoundException(tableName); + } else { + return partitions; + } + }

            +
            + @Override
            + public List<PartitionDescProto> getPartitionsByAlgebra(PartitionsByAlgebraProto request) throws
            + UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException,
            + PartitionNotFoundException, UnsupportedException

            Unknown macro: { + + List<PartitionDescProto> list = null; + + try { + String databaseName = request.getDatabaseName(); + String tableName = request.getTableName(); + + TableDescProto tableDesc = getTable(databaseName, tableName); + String filter = getFilter(databaseName, tableName, tableDesc.getPartition().getExpressionSchema().getFieldsList() + , request.getAlgebra()); + list = getPartitionsByFilterFromHiveMetaStore(databaseName, tableName, filter); + } catch (UnsupportedException ue) { + throw ue; + } catch (Exception se) { + throw new TajoInternalError(se); + } + + return list; + }

            +
            + private String getFilter(String databaseName, String tableName, List<ColumnProto> partitionColumns
            + , String json) throws TajoException {
            +
            + Expr[] exprs = null;
            +
            + if (json != null && !json.isEmpty())

            { + Expr algebra = JsonHelper.fromJson(json, Expr.class); + exprs = AlgebraicUtil.toConjunctiveNormalFormArray(algebra); + }

            +
            + PartitionFilterAlgebraVisitor visitor = new PartitionFilterAlgebraVisitor();
            + visitor.setIsHiveCatalog(true);
            +
            + Expr[] filters = AlgebraicUtil.getAccumulatedFiltersByExpr(databaseName + "." + tableName, partitionColumns, exprs);
            +
            + StringBuffer sb = new StringBuffer();
            +
            + // Write join clause from second column to last column.
            + Column target;
            +
            + int addedFilter = 0;
            + String result;
            + for (int i = 0; i < partitionColumns.size(); i++) {
            + target = new Column(partitionColumns.get);
            +
            + if (!(filters[i] instanceof IsNullPredicate)) {
            + visitor.setColumn(target);
            + visitor.visit(null, new Stack<Expr>(), filters[i]);
            + result = visitor.getResult();
            +
            + // If visitor build filter successfully, add filter to be used for executing hive api.
            + if (result.length() > 0) {

              • End diff –

          When does the result have zero-length?
          We should handle this situation.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r39943564 — Diff: tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogStore.java — @@ -845,11 +856,155 @@ public boolean existPartitionMethod(String databaseName, String tableName) throw } @Override public List<CatalogProtos.PartitionDescProto> getPartitions(String databaseName, String tableName) { throw new UnsupportedOperationException(); + public List<CatalogProtos.PartitionDescProto> getAllPartitions(String databaseName, String tableName) + throws UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException, + PartitionNotFoundException Unknown macro: { + PartitionsByFilterProto.Builder request = PartitionsByFilterProto.newBuilder(); + request.setDatabaseName(databaseName); + request.setTableName(tableName); + request.setFilter(""); + + List<PartitionDescProto> partitions = getPartitionsByFilter(request.build()); + if (partitions.size() == 0) { + throw new PartitionNotFoundException(tableName); + } else { + return partitions; + } + } + + @Override + public List<PartitionDescProto> getPartitionsByAlgebra(PartitionsByAlgebraProto request) throws + UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException, + PartitionNotFoundException, UnsupportedException Unknown macro: { + + List<PartitionDescProto> list = null; + + try { + String databaseName = request.getDatabaseName(); + String tableName = request.getTableName(); + + TableDescProto tableDesc = getTable(databaseName, tableName); + String filter = getFilter(databaseName, tableName, tableDesc.getPartition().getExpressionSchema().getFieldsList() + , request.getAlgebra()); + list = getPartitionsByFilterFromHiveMetaStore(databaseName, tableName, filter); + } catch (UnsupportedException ue) { + throw ue; + } catch (Exception se) { + throw new TajoInternalError(se); + } + + return list; + } + + private String getFilter(String databaseName, String tableName, List<ColumnProto> partitionColumns + , String json) throws TajoException { + + Expr[] exprs = null; + + if (json != null && !json.isEmpty()) { + Expr algebra = JsonHelper.fromJson(json, Expr.class); + exprs = AlgebraicUtil.toConjunctiveNormalFormArray(algebra); + } + + PartitionFilterAlgebraVisitor visitor = new PartitionFilterAlgebraVisitor(); + visitor.setIsHiveCatalog(true); + + Expr[] filters = AlgebraicUtil.getAccumulatedFiltersByExpr(databaseName + "." + tableName, partitionColumns, exprs); + + StringBuffer sb = new StringBuffer(); + + // Write join clause from second column to last column. + Column target; + + int addedFilter = 0; + String result; + for (int i = 0; i < partitionColumns.size(); i++) { + target = new Column(partitionColumns.get ); + + if (!(filters [i] instanceof IsNullPredicate)) { + visitor.setColumn(target); + visitor.visit(null, new Stack<Expr>(), filters [i] ); + result = visitor.getResult(); + + // If visitor build filter successfully, add filter to be used for executing hive api. + if (result.length() > 0) { End diff – When does the result have zero-length? We should handle this situation.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r39943957

          — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java —
          @@ -1043,6 +1043,90 @@ public GetTablePartitionsResponse getAllPartitions(RpcController controller, Nul
          }

          @Override
          + public GetPartitionsResponse getPartitionsByAlgebra(RpcController controller,
          + PartitionsByAlgebraProto request) throws ServiceException {
          + String dbName = request.getDatabaseName();
          + String tbName = request.getTableName();
          +
          + try {
          + // linked meta data do not support partition.
          + // So, the request that wants to get partitions in this db will be failed.
          + if (linkedMetadataManager.existsDatabase(dbName)) {
          — End diff –

          These codes can be much simplified. Please refer to the ```existsTable()``` method of this class.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r39943957 — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java — @@ -1043,6 +1043,90 @@ public GetTablePartitionsResponse getAllPartitions(RpcController controller, Nul } @Override + public GetPartitionsResponse getPartitionsByAlgebra(RpcController controller, + PartitionsByAlgebraProto request) throws ServiceException { + String dbName = request.getDatabaseName(); + String tbName = request.getTableName(); + + try { + // linked meta data do not support partition. + // So, the request that wants to get partitions in this db will be failed. + if (linkedMetadataManager.existsDatabase(dbName)) { — End diff – These codes can be much simplified. Please refer to the ```existsTable()``` method of this class.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r39943983

          — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java —
          @@ -1043,6 +1043,90 @@ public GetTablePartitionsResponse getAllPartitions(RpcController controller, Nul
          }

          @Override
          + public GetPartitionsResponse getPartitionsByAlgebra(RpcController controller,
          + PartitionsByAlgebraProto request) throws ServiceException {
          + String dbName = request.getDatabaseName();
          + String tbName = request.getTableName();
          +
          + try {
          + // linked meta data do not support partition.
          + // So, the request that wants to get partitions in this db will be failed.
          + if (linkedMetadataManager.existsDatabase(dbName))

          { + return GetPartitionsResponse.newBuilder().setState(errUndefinedPartitionMethod(tbName)).build(); + }

          + } catch (Throwable t)

          { + printStackTraceIfError(LOG, t); + return GetPartitionsResponse.newBuilder() + .setState(returnError(t)) + .build(); + }

          +
          + if (metaDictionary.isSystemDatabase(dbName))

          { + return GetPartitionsResponse.newBuilder().setState(errUndefinedPartitionMethod(tbName)).build(); + }

          +
          + rlock.lock();
          + try

          { + GetPartitionsResponse.Builder builder = GetPartitionsResponse.newBuilder(); + List<PartitionDescProto> partitions = store.getPartitionsByAlgebra(request); + builder.addAllPartition(partitions); + builder.setState(OK); + return builder.build(); + }

          catch (Throwable t)

          { + printStackTraceIfError(LOG, t); + + return GetPartitionsResponse.newBuilder() + .setState(returnError(t)) + .build(); + + }

          finally

          { + rlock.unlock(); + }

          + }
          +
          + @Override
          + public GetPartitionsResponse getPartitionsByFilter(RpcController controller,
          + PartitionsByFilterProto request) throws ServiceException {
          + String dbName = request.getDatabaseName();
          + String tbName = request.getTableName();
          +
          + try {
          + // linked meta data do not support partition.
          + // So, the request that wants to get partitions in this db will be failed.
          + if (linkedMetadataManager.existsDatabase(dbName)) {
          — End diff –

          These codes can be much simplified. Please refer to the ```existsTable()``` method of this class.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r39943983 — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java — @@ -1043,6 +1043,90 @@ public GetTablePartitionsResponse getAllPartitions(RpcController controller, Nul } @Override + public GetPartitionsResponse getPartitionsByAlgebra(RpcController controller, + PartitionsByAlgebraProto request) throws ServiceException { + String dbName = request.getDatabaseName(); + String tbName = request.getTableName(); + + try { + // linked meta data do not support partition. + // So, the request that wants to get partitions in this db will be failed. + if (linkedMetadataManager.existsDatabase(dbName)) { + return GetPartitionsResponse.newBuilder().setState(errUndefinedPartitionMethod(tbName)).build(); + } + } catch (Throwable t) { + printStackTraceIfError(LOG, t); + return GetPartitionsResponse.newBuilder() + .setState(returnError(t)) + .build(); + } + + if (metaDictionary.isSystemDatabase(dbName)) { + return GetPartitionsResponse.newBuilder().setState(errUndefinedPartitionMethod(tbName)).build(); + } + + rlock.lock(); + try { + GetPartitionsResponse.Builder builder = GetPartitionsResponse.newBuilder(); + List<PartitionDescProto> partitions = store.getPartitionsByAlgebra(request); + builder.addAllPartition(partitions); + builder.setState(OK); + return builder.build(); + } catch (Throwable t) { + printStackTraceIfError(LOG, t); + + return GetPartitionsResponse.newBuilder() + .setState(returnError(t)) + .build(); + + } finally { + rlock.unlock(); + } + } + + @Override + public GetPartitionsResponse getPartitionsByFilter(RpcController controller, + PartitionsByFilterProto request) throws ServiceException { + String dbName = request.getDatabaseName(); + String tbName = request.getTableName(); + + try { + // linked meta data do not support partition. + // So, the request that wants to get partitions in this db will be failed. + if (linkedMetadataManager.existsDatabase(dbName)) { — End diff – These codes can be much simplified. Please refer to the ```existsTable()``` method of this class.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r39944135

          — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java —
          @@ -2147,6 +2154,8 @@ private void ensurePartitionTable(String tbName, int tableId)
          builder.setId(res.getInt(COL_PARTITIONS_PK));
          builder.setPath(res.getString("PATH"));
          builder.setPartitionName(partitionName);
          + builder.setNumBytes(res.getLong(COL_PARTITION_BYTES));
          + builder.setNumFiles(res.getLong(COL_PARTITION_FILES));
          — End diff –

          I wonder why number of files in a partition is required.
          Can you tell me an example?

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r39944135 — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java — @@ -2147,6 +2154,8 @@ private void ensurePartitionTable(String tbName, int tableId) builder.setId(res.getInt(COL_PARTITIONS_PK)); builder.setPath(res.getString("PATH")); builder.setPartitionName(partitionName); + builder.setNumBytes(res.getLong(COL_PARTITION_BYTES)); + builder.setNumFiles(res.getLong(COL_PARTITION_FILES)); — End diff – I wonder why number of files in a partition is required. Can you tell me an example?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r39944228

          — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java —
          @@ -2228,6 +2244,257 @@ private void setPartitionKeys(int pid, PartitionDescProto.Builder partitionDesc)
          return partitions;
          }

          + /**
          + * Check if list of partitions exist on catalog.
          + *
          + *
          + * @param databaseId
          + * @param tableId
          + * @return
          + */
          + public boolean existPartitionsOnCatalog(int tableId) {
          + Connection conn = null;
          + ResultSet res = null;
          + PreparedStatement pstmt = null;
          + boolean result = false;
          +
          + try {
          + String sql = "SELECT COUNT CNT FROM "
          — End diff –

          This is trivial, but you can perform a simple select rather than count aggregation which is more expensive operation.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r39944228 — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java — @@ -2228,6 +2244,257 @@ private void setPartitionKeys(int pid, PartitionDescProto.Builder partitionDesc) return partitions; } + /** + * Check if list of partitions exist on catalog. + * + * + * @param databaseId + * @param tableId + * @return + */ + public boolean existPartitionsOnCatalog(int tableId) { + Connection conn = null; + ResultSet res = null; + PreparedStatement pstmt = null; + boolean result = false; + + try { + String sql = "SELECT COUNT CNT FROM " — End diff – This is trivial, but you can perform a simple select rather than count aggregation which is more expensive operation.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r39944289

          — Diff: tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogStore.java —
          @@ -845,11 +856,155 @@ public boolean existPartitionMethod(String databaseName, String tableName) throw
          }

          @Override

          • public List<CatalogProtos.PartitionDescProto> getPartitions(String databaseName,
          • String tableName) {
          • throw new UnsupportedOperationException();
            + public List<CatalogProtos.PartitionDescProto> getAllPartitions(String databaseName, String tableName)
            + throws UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException,
            + PartitionNotFoundException
            Unknown macro: { + PartitionsByFilterProto.Builder request = PartitionsByFilterProto.newBuilder(); + request.setDatabaseName(databaseName); + request.setTableName(tableName); + request.setFilter(""); + + List<PartitionDescProto> partitions = getPartitionsByFilter(request.build()); + if (partitions.size() == 0) { + throw new PartitionNotFoundException(tableName); + } else { + return partitions; + } + }

            +
            + @Override
            + public List<PartitionDescProto> getPartitionsByAlgebra(PartitionsByAlgebraProto request) throws
            + UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException,
            + PartitionNotFoundException, UnsupportedException {
            +
            + List<PartitionDescProto> list = null;
            +
            + try

            { + String databaseName = request.getDatabaseName(); + String tableName = request.getTableName(); + + TableDescProto tableDesc = getTable(databaseName, tableName); + String filter = getFilter(databaseName, tableName, tableDesc.getPartition().getExpressionSchema().getFieldsList() + , request.getAlgebra()); + list = getPartitionsByFilterFromHiveMetaStore(databaseName, tableName, filter); + }

            catch (UnsupportedException ue) {

              • End diff –

          I found it.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r39944289 — Diff: tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogStore.java — @@ -845,11 +856,155 @@ public boolean existPartitionMethod(String databaseName, String tableName) throw } @Override public List<CatalogProtos.PartitionDescProto> getPartitions(String databaseName, String tableName) { throw new UnsupportedOperationException(); + public List<CatalogProtos.PartitionDescProto> getAllPartitions(String databaseName, String tableName) + throws UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException, + PartitionNotFoundException Unknown macro: { + PartitionsByFilterProto.Builder request = PartitionsByFilterProto.newBuilder(); + request.setDatabaseName(databaseName); + request.setTableName(tableName); + request.setFilter(""); + + List<PartitionDescProto> partitions = getPartitionsByFilter(request.build()); + if (partitions.size() == 0) { + throw new PartitionNotFoundException(tableName); + } else { + return partitions; + } + } + + @Override + public List<PartitionDescProto> getPartitionsByAlgebra(PartitionsByAlgebraProto request) throws + UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException, + PartitionNotFoundException, UnsupportedException { + + List<PartitionDescProto> list = null; + + try { + String databaseName = request.getDatabaseName(); + String tableName = request.getTableName(); + + TableDescProto tableDesc = getTable(databaseName, tableName); + String filter = getFilter(databaseName, tableName, tableDesc.getPartition().getExpressionSchema().getFieldsList() + , request.getAlgebra()); + list = getPartitionsByFilterFromHiveMetaStore(databaseName, tableName, filter); + } catch (UnsupportedException ue) { End diff – I found it.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r39944352

          — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/CatalogStore.java —
          @@ -97,13 +97,31 @@ boolean existPartitionMethod(String databaseName, String tableName) throws Undef

          • @return
          • @throws TajoException
            */
          • List<CatalogProtos.PartitionDescProto> getPartitions(String databaseName, String tableName) throws
          • UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException;
            + List<CatalogProtos.PartitionDescProto> getAllPartitions(String databaseName, String tableName) throws
            + UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException,
            + PartitionNotFoundException, UnsupportedException;

          CatalogProtos.PartitionDescProto getPartition(String databaseName, String tableName,

          • String partitionName)
          • throws UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionException,
          • UndefinedPartitionMethodException;
            + String partitionName) throws UndefinedDatabaseException,
            + UndefinedTableException, UndefinedPartitionMethodException, UndefinedPartitionException;
            +
            + /**
            + * PartitionedTableRewriter take a look into partition directories for rewriting filter conditions. But if there
            + * are lots of sub directories on HDFS, such as, more than 10,000 directories,
            + * it might be cause overload to NameNode. Thus, CatalogStore need to provide partition directories for specified
            + * filter conditions. This scan right partition directories on CatalogStore with where clause.
            + *
            + * @param request contains database name, table name, algebra expressions, parameter for executing PrepareStatement
            + * @return list of TablePartitionProto
            + * @throws UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException
            + */
            + List<PartitionDescProto> getPartitionsByAlgebra(PartitionsByAlgebraProto request) throws
            + UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException,
            + UndefinedOperatorException, PartitionNotFoundException, UnsupportedException;
            +
            + List<PartitionDescProto> getPartitionsByFilter(PartitionsByFilterProto request) throws
              • End diff –

          You need to add a description for this method.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r39944352 — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/CatalogStore.java — @@ -97,13 +97,31 @@ boolean existPartitionMethod(String databaseName, String tableName) throws Undef @return @throws TajoException */ List<CatalogProtos.PartitionDescProto> getPartitions(String databaseName, String tableName) throws UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException; + List<CatalogProtos.PartitionDescProto> getAllPartitions(String databaseName, String tableName) throws + UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException, + PartitionNotFoundException, UnsupportedException; CatalogProtos.PartitionDescProto getPartition(String databaseName, String tableName, String partitionName) throws UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionException, UndefinedPartitionMethodException; + String partitionName) throws UndefinedDatabaseException, + UndefinedTableException, UndefinedPartitionMethodException, UndefinedPartitionException; + + /** + * PartitionedTableRewriter take a look into partition directories for rewriting filter conditions. But if there + * are lots of sub directories on HDFS, such as, more than 10,000 directories, + * it might be cause overload to NameNode. Thus, CatalogStore need to provide partition directories for specified + * filter conditions. This scan right partition directories on CatalogStore with where clause. + * + * @param request contains database name, table name, algebra expressions, parameter for executing PrepareStatement + * @return list of TablePartitionProto + * @throws UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException + */ + List<PartitionDescProto> getPartitionsByAlgebra(PartitionsByAlgebraProto request) throws + UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException, + UndefinedOperatorException, PartitionNotFoundException, UnsupportedException; + + List<PartitionDescProto> getPartitionsByFilter(PartitionsByFilterProto request) throws End diff – You need to add a description for this method.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r39944471

          — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java —
          @@ -2228,6 +2244,257 @@ private void setPartitionKeys(int pid, PartitionDescProto.Builder partitionDesc)
          return partitions;
          }

          + /**
          + * Check if list of partitions exist on catalog.
          + *
          + *
          + * @param databaseId
          + * @param tableId
          + * @return
          + */
          + public boolean existPartitionsOnCatalog(int tableId) {
          + Connection conn = null;
          + ResultSet res = null;
          + PreparedStatement pstmt = null;
          + boolean result = false;
          +
          + try {
          + String sql = "SELECT COUNT CNT FROM "
          + + TB_PARTTIONS +" WHERE " + COL_TABLES_PK + " = ? ";
          +
          + if (LOG.isDebugEnabled())

          { + LOG.debug(sql); + }

          +
          + conn = getConnection();
          + pstmt = conn.prepareStatement(sql);
          + pstmt.setInt(1, tableId);
          + res = pstmt.executeQuery();
          +
          + if (res.next()) {
          + if (res.getInt("CNT") > 0)

          { + result = true; + }

          + }
          + } catch (SQLException se)

          { + throw new TajoInternalError(se); + }

          finally

          { + CatalogUtil.closeQuietly(pstmt, res); + }

          + return result;
          + }
          +
          + @Override
          + public List<PartitionDescProto> getPartitionsByFilter(PartitionsByFilterProto request) {
          + throw new TajoRuntimeException(new UnsupportedException());
          — End diff –

          ```UnsupportedException``` means that we don't support this feature. But, this is not true because we support partition pruning with simple filters as well as more complex conditions.
          So, it would be good if this method just call ```getPartitionsByAlgebra()``` in the below.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r39944471 — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java — @@ -2228,6 +2244,257 @@ private void setPartitionKeys(int pid, PartitionDescProto.Builder partitionDesc) return partitions; } + /** + * Check if list of partitions exist on catalog. + * + * + * @param databaseId + * @param tableId + * @return + */ + public boolean existPartitionsOnCatalog(int tableId) { + Connection conn = null; + ResultSet res = null; + PreparedStatement pstmt = null; + boolean result = false; + + try { + String sql = "SELECT COUNT CNT FROM " + + TB_PARTTIONS +" WHERE " + COL_TABLES_PK + " = ? "; + + if (LOG.isDebugEnabled()) { + LOG.debug(sql); + } + + conn = getConnection(); + pstmt = conn.prepareStatement(sql); + pstmt.setInt(1, tableId); + res = pstmt.executeQuery(); + + if (res.next()) { + if (res.getInt("CNT") > 0) { + result = true; + } + } + } catch (SQLException se) { + throw new TajoInternalError(se); + } finally { + CatalogUtil.closeQuietly(pstmt, res); + } + return result; + } + + @Override + public List<PartitionDescProto> getPartitionsByFilter(PartitionsByFilterProto request) { + throw new TajoRuntimeException(new UnsupportedException()); — End diff – ```UnsupportedException``` means that we don't support this feature. But, this is not true because we support partition pruning with simple filters as well as more complex conditions. So, it would be good if this method just call ```getPartitionsByAlgebra()``` in the below.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user blrunner commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r39944600

          — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java —
          @@ -2147,6 +2154,8 @@ private void ensurePartitionTable(String tbName, int tableId)
          builder.setId(res.getInt(COL_PARTITIONS_PK));
          builder.setPath(res.getString("PATH"));
          builder.setPartitionName(partitionName);
          + builder.setNumBytes(res.getLong(COL_PARTITION_BYTES));
          + builder.setNumFiles(res.getLong(COL_PARTITION_FILES));
          — End diff –

          Honestly, I added num of bytes and number of files by reference to PartitionedTableRewriter::updateTableStat. But now, it looks like unnecessary information.

          Show
          githubbot ASF GitHub Bot added a comment - Github user blrunner commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r39944600 — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java — @@ -2147,6 +2154,8 @@ private void ensurePartitionTable(String tbName, int tableId) builder.setId(res.getInt(COL_PARTITIONS_PK)); builder.setPath(res.getString("PATH")); builder.setPartitionName(partitionName); + builder.setNumBytes(res.getLong(COL_PARTITION_BYTES)); + builder.setNumFiles(res.getLong(COL_PARTITION_FILES)); — End diff – Honestly, I added num of bytes and number of files by reference to PartitionedTableRewriter::updateTableStat. But now, it looks like unnecessary information.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r39944723

          — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java —
          @@ -2228,6 +2244,257 @@ private void setPartitionKeys(int pid, PartitionDescProto.Builder partitionDesc)
          return partitions;
          }

          + /**
          + * Check if list of partitions exist on catalog.
          + *
          + *
          + * @param databaseId
          + * @param tableId
          + * @return
          + */
          + public boolean existPartitionsOnCatalog(int tableId) {
          + Connection conn = null;
          + ResultSet res = null;
          + PreparedStatement pstmt = null;
          + boolean result = false;
          +
          + try {
          + String sql = "SELECT COUNT CNT FROM "
          + + TB_PARTTIONS +" WHERE " + COL_TABLES_PK + " = ? ";
          +
          + if (LOG.isDebugEnabled())

          { + LOG.debug(sql); + }

          +
          + conn = getConnection();
          + pstmt = conn.prepareStatement(sql);
          + pstmt.setInt(1, tableId);
          + res = pstmt.executeQuery();
          +
          + if (res.next()) {
          + if (res.getInt("CNT") > 0)

          { + result = true; + }

          + }
          + } catch (SQLException se)

          { + throw new TajoInternalError(se); + }

          finally

          { + CatalogUtil.closeQuietly(pstmt, res); + }

          + return result;
          + }
          +
          + @Override
          + public List<PartitionDescProto> getPartitionsByFilter(PartitionsByFilterProto request)

          { + throw new TajoRuntimeException(new UnsupportedException()); + }

          +
          + @Override
          + public List<PartitionDescProto> getPartitionsByAlgebra(PartitionsByAlgebraProto request) throws
          + UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException,
          + PartitionNotFoundException, UnsupportedException {
          + Connection conn = null;
          + PreparedStatement pstmt = null;
          + ResultSet res = null;
          + int currentIndex = 1;
          + String selectStatement = null;
          +
          + List<PartitionDescProto> partitions = TUtil.newList();
          + List<PartitionFilterSet> filterSets = TUtil.newList();
          +
          + int databaseId = getDatabaseId(request.getDatabaseName());
          + int tableId = getTableId(databaseId, request.getDatabaseName(), request.getTableName());
          + if (!existPartitionMethod(request.getDatabaseName(), request.getTableName()))

          { + throw new UndefinedPartitionMethodException(request.getTableName()); + }

          +
          + if (!existPartitionsOnCatalog(tableId))

          { + throw new PartitionNotFoundException(request.getTableName()); + }

          +
          + try {
          + TableDescProto tableDesc = getTable(request.getDatabaseName(), request.getTableName());
          +
          + selectStatement = getSelectStatementForPartitions(tableDesc.getTableName(), tableDesc.getPartition()
          + .getExpressionSchema().getFieldsList(), request.getAlgebra(), filterSets);
          +
          + conn = getConnection();
          + pstmt = conn.prepareStatement(selectStatement);
          +
          + // Set table id by force because first parameter of all direct sql is table id
          + pstmt.setInt(currentIndex, tableId);
          + currentIndex++;
          +
          + for (PartitionFilterSet filter : filterSets) {
          + // Set table id by force because all filters have table id as first parameter.
          + pstmt.setInt(currentIndex, tableId);
          + currentIndex++;
          +
          + for (Pair<Type, Object> parameter : filter.getParameters()) {
          + switch (parameter.getFirst())

          { + case BOOLEAN: + pstmt.setBoolean(currentIndex, (Boolean)parameter.getSecond()); + break; + case INT8: + pstmt.setLong(currentIndex, (Long) parameter.getSecond()); + break; + case FLOAT8: + pstmt.setDouble(currentIndex, (Double) parameter.getSecond()); + break; + case DATE: + pstmt.setDate(currentIndex, (Date) parameter.getSecond()); + break; + case TIMESTAMP: + pstmt.setTimestamp(currentIndex, (Timestamp) parameter.getSecond()); + break; + case TIME: + pstmt.setTime(currentIndex, (Time) parameter.getSecond()); + break; + default: + pstmt.setString(currentIndex, (String) parameter.getSecond()); + break; + }

          + currentIndex++;
          + }
          + }
          +
          + res = pstmt.executeQuery();
          +
          + while (res.next())

          { + PartitionDescProto.Builder builder = PartitionDescProto.newBuilder(); + + builder.setId(res.getInt(COL_PARTITIONS_PK)); + builder.setPartitionName(res.getString("PARTITION_NAME")); + builder.setPath(res.getString("PATH")); + builder.setNumBytes(res.getLong(COL_PARTITION_BYTES)); + builder.setNumFiles(res.getLong(COL_PARTITION_FILES)); + + partitions.add(builder.build()); + }

          + } catch (TajoException se) {
          + throw new TajoInternalError(se);
          — End diff –

          All UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException, and PartitionNotFoundException exceptions are subclass of TajoException. This code will prevent these exceptions from propagating properly.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r39944723 — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java — @@ -2228,6 +2244,257 @@ private void setPartitionKeys(int pid, PartitionDescProto.Builder partitionDesc) return partitions; } + /** + * Check if list of partitions exist on catalog. + * + * + * @param databaseId + * @param tableId + * @return + */ + public boolean existPartitionsOnCatalog(int tableId) { + Connection conn = null; + ResultSet res = null; + PreparedStatement pstmt = null; + boolean result = false; + + try { + String sql = "SELECT COUNT CNT FROM " + + TB_PARTTIONS +" WHERE " + COL_TABLES_PK + " = ? "; + + if (LOG.isDebugEnabled()) { + LOG.debug(sql); + } + + conn = getConnection(); + pstmt = conn.prepareStatement(sql); + pstmt.setInt(1, tableId); + res = pstmt.executeQuery(); + + if (res.next()) { + if (res.getInt("CNT") > 0) { + result = true; + } + } + } catch (SQLException se) { + throw new TajoInternalError(se); + } finally { + CatalogUtil.closeQuietly(pstmt, res); + } + return result; + } + + @Override + public List<PartitionDescProto> getPartitionsByFilter(PartitionsByFilterProto request) { + throw new TajoRuntimeException(new UnsupportedException()); + } + + @Override + public List<PartitionDescProto> getPartitionsByAlgebra(PartitionsByAlgebraProto request) throws + UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException, + PartitionNotFoundException, UnsupportedException { + Connection conn = null; + PreparedStatement pstmt = null; + ResultSet res = null; + int currentIndex = 1; + String selectStatement = null; + + List<PartitionDescProto> partitions = TUtil.newList(); + List<PartitionFilterSet> filterSets = TUtil.newList(); + + int databaseId = getDatabaseId(request.getDatabaseName()); + int tableId = getTableId(databaseId, request.getDatabaseName(), request.getTableName()); + if (!existPartitionMethod(request.getDatabaseName(), request.getTableName())) { + throw new UndefinedPartitionMethodException(request.getTableName()); + } + + if (!existPartitionsOnCatalog(tableId)) { + throw new PartitionNotFoundException(request.getTableName()); + } + + try { + TableDescProto tableDesc = getTable(request.getDatabaseName(), request.getTableName()); + + selectStatement = getSelectStatementForPartitions(tableDesc.getTableName(), tableDesc.getPartition() + .getExpressionSchema().getFieldsList(), request.getAlgebra(), filterSets); + + conn = getConnection(); + pstmt = conn.prepareStatement(selectStatement); + + // Set table id by force because first parameter of all direct sql is table id + pstmt.setInt(currentIndex, tableId); + currentIndex++; + + for (PartitionFilterSet filter : filterSets) { + // Set table id by force because all filters have table id as first parameter. + pstmt.setInt(currentIndex, tableId); + currentIndex++; + + for (Pair<Type, Object> parameter : filter.getParameters()) { + switch (parameter.getFirst()) { + case BOOLEAN: + pstmt.setBoolean(currentIndex, (Boolean)parameter.getSecond()); + break; + case INT8: + pstmt.setLong(currentIndex, (Long) parameter.getSecond()); + break; + case FLOAT8: + pstmt.setDouble(currentIndex, (Double) parameter.getSecond()); + break; + case DATE: + pstmt.setDate(currentIndex, (Date) parameter.getSecond()); + break; + case TIMESTAMP: + pstmt.setTimestamp(currentIndex, (Timestamp) parameter.getSecond()); + break; + case TIME: + pstmt.setTime(currentIndex, (Time) parameter.getSecond()); + break; + default: + pstmt.setString(currentIndex, (String) parameter.getSecond()); + break; + } + currentIndex++; + } + } + + res = pstmt.executeQuery(); + + while (res.next()) { + PartitionDescProto.Builder builder = PartitionDescProto.newBuilder(); + + builder.setId(res.getInt(COL_PARTITIONS_PK)); + builder.setPartitionName(res.getString("PARTITION_NAME")); + builder.setPath(res.getString("PATH")); + builder.setNumBytes(res.getLong(COL_PARTITION_BYTES)); + builder.setNumFiles(res.getLong(COL_PARTITION_FILES)); + + partitions.add(builder.build()); + } + } catch (TajoException se) { + throw new TajoInternalError(se); — End diff – All UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException, and PartitionNotFoundException exceptions are subclass of TajoException. This code will prevent these exceptions from propagating properly.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r39944783

          — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java —
          @@ -2228,6 +2244,257 @@ private void setPartitionKeys(int pid, PartitionDescProto.Builder partitionDesc)
          return partitions;
          }

          + /**
          + * Check if list of partitions exist on catalog.
          + *
          + *
          + * @param databaseId
          + * @param tableId
          + * @return
          + */
          + public boolean existPartitionsOnCatalog(int tableId) {
          + Connection conn = null;
          + ResultSet res = null;
          + PreparedStatement pstmt = null;
          + boolean result = false;
          +
          + try {
          + String sql = "SELECT COUNT CNT FROM "
          + + TB_PARTTIONS +" WHERE " + COL_TABLES_PK + " = ? ";
          +
          + if (LOG.isDebugEnabled())

          { + LOG.debug(sql); + }

          +
          + conn = getConnection();
          + pstmt = conn.prepareStatement(sql);
          + pstmt.setInt(1, tableId);
          + res = pstmt.executeQuery();
          +
          + if (res.next()) {
          + if (res.getInt("CNT") > 0)

          { + result = true; + }

          + }
          + } catch (SQLException se)

          { + throw new TajoInternalError(se); + } finally { + CatalogUtil.closeQuietly(pstmt, res); + }
          + return result;
          + }
          +
          + @Override
          + public List<PartitionDescProto> getPartitionsByFilter(PartitionsByFilterProto request) { + throw new TajoRuntimeException(new UnsupportedException()); + }
          +
          + @Override
          + public List<PartitionDescProto> getPartitionsByAlgebra(PartitionsByAlgebraProto request) throws
          + UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException,
          + PartitionNotFoundException, UnsupportedException {
          + Connection conn = null;
          + PreparedStatement pstmt = null;
          + ResultSet res = null;
          + int currentIndex = 1;
          + String selectStatement = null;
          +
          + List<PartitionDescProto> partitions = TUtil.newList();
          + List<PartitionFilterSet> filterSets = TUtil.newList();
          +
          + int databaseId = getDatabaseId(request.getDatabaseName());
          + int tableId = getTableId(databaseId, request.getDatabaseName(), request.getTableName());
          + if (!existPartitionMethod(request.getDatabaseName(), request.getTableName())) { + throw new UndefinedPartitionMethodException(request.getTableName()); + }
          +
          + if (!existPartitionsOnCatalog(tableId)) { + throw new PartitionNotFoundException(request.getTableName()); + }
          +
          + try {
          + TableDescProto tableDesc = getTable(request.getDatabaseName(), request.getTableName());
          +
          + selectStatement = getSelectStatementForPartitions(tableDesc.getTableName(), tableDesc.getPartition()
          + .getExpressionSchema().getFieldsList(), request.getAlgebra(), filterSets);
          +
          + conn = getConnection();
          + pstmt = conn.prepareStatement(selectStatement);
          +
          + // Set table id by force because first parameter of all direct sql is table id
          + pstmt.setInt(currentIndex, tableId);
          + currentIndex++;
          +
          + for (PartitionFilterSet filter : filterSets) {
          + // Set table id by force because all filters have table id as first parameter.
          + pstmt.setInt(currentIndex, tableId);
          + currentIndex++;
          +
          + for (Pair<Type, Object> parameter : filter.getParameters()) {
          + switch (parameter.getFirst()) { + case BOOLEAN: + pstmt.setBoolean(currentIndex, (Boolean)parameter.getSecond()); + break; + case INT8: + pstmt.setLong(currentIndex, (Long) parameter.getSecond()); + break; + case FLOAT8: + pstmt.setDouble(currentIndex, (Double) parameter.getSecond()); + break; + case DATE: + pstmt.setDate(currentIndex, (Date) parameter.getSecond()); + break; + case TIMESTAMP: + pstmt.setTimestamp(currentIndex, (Timestamp) parameter.getSecond()); + break; + case TIME: + pstmt.setTime(currentIndex, (Time) parameter.getSecond()); + break; + default: + pstmt.setString(currentIndex, (String) parameter.getSecond()); + break; + }
          + currentIndex++;
          + }
          + }
          +
          + res = pstmt.executeQuery();
          +
          + while (res.next()) { + PartitionDescProto.Builder builder = PartitionDescProto.newBuilder(); + + builder.setId(res.getInt(COL_PARTITIONS_PK)); + builder.setPartitionName(res.getString("PARTITION_NAME")); + builder.setPath(res.getString("PATH")); + builder.setNumBytes(res.getLong(COL_PARTITION_BYTES)); + builder.setNumFiles(res.getLong(COL_PARTITION_FILES)); + + partitions.add(builder.build()); + }
          + } catch (TajoException se) { + throw new TajoInternalError(se); + }

          catch (SQLException se)

          { + throw new TajoInternalError(se); + }

          finally

          { + CatalogUtil.closeQuietly(pstmt, res); + }

          +
          + return partitions;
          + }
          +
          + /**
          + * Create a select statement and parameters for querying partitions and partition keys in CatalogStore.
          + *
          + * For example, consider you have a partitioned table for three columns (i.e., col1, col2, col3).
          + * Assume that an user gives a condition WHERE (col1 ='1' or col1 = '100') and col3 > 20.
          + * There is no filter condition corresponding to col2.
          + *
          + * Then, the sql would be generated as following:
          + *
          + * SELECT A.PARTITION_ID, A.PARTITION_NAME, A.PATH FROM PARTITIONS A
          + * WHERE A.TID = ?
          + * AND A.PARTITION_ID IN (
          + * SELECT T1.PARTITION_ID FROM PARTITION_KEYS T1
          + * JOIN PARTITION_KEYS T2 ON T1.TID=T2.TID AND T1.PARTITION_ID = T2.PARTITION_ID AND T2.TID = ?
          + * AND ( T2.COLUMN_NAME = 'col2' AND T2.PARTITION_VALUE IS NOT NULL )
          + * JOIN PARTITION_KEYS T3 ON T1.TID=T3.TID AND T1.PARTITION_ID = T3.PARTITION_ID AND T3.TID = ?
          + * AND ( T3.COLUMN_NAME = 'col3' AND T3.PARTITION_VALUE > ? )
          + * WHERE T1.TID = ? AND ( T1.COLUMN_NAME = 'col1' AND T1.PARTITION_VALUE = ? )
          + * OR ( T1.COLUMN_NAME = 'col1' AND T1.PARTITION_VALUE = ? )
          + )
          + *
          + * @param partitionColumns
          + * @param json
          + * @param filterSets
          + * @return
          + * @throws TajoException
          + * @throws SQLException
          + */
          + private String getSelectStatementForPartitions(String tableName, List<ColumnProto> partitionColumns, String json,
          + List<PartitionFilterSet> filterSets) throws TajoException {
          — End diff –

          At least, you should specify that ```filterSets``` is one of the results of this method.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r39944783 — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java — @@ -2228,6 +2244,257 @@ private void setPartitionKeys(int pid, PartitionDescProto.Builder partitionDesc) return partitions; } + /** + * Check if list of partitions exist on catalog. + * + * + * @param databaseId + * @param tableId + * @return + */ + public boolean existPartitionsOnCatalog(int tableId) { + Connection conn = null; + ResultSet res = null; + PreparedStatement pstmt = null; + boolean result = false; + + try { + String sql = "SELECT COUNT CNT FROM " + + TB_PARTTIONS +" WHERE " + COL_TABLES_PK + " = ? "; + + if (LOG.isDebugEnabled()) { + LOG.debug(sql); + } + + conn = getConnection(); + pstmt = conn.prepareStatement(sql); + pstmt.setInt(1, tableId); + res = pstmt.executeQuery(); + + if (res.next()) { + if (res.getInt("CNT") > 0) { + result = true; + } + } + } catch (SQLException se) { + throw new TajoInternalError(se); + } finally { + CatalogUtil.closeQuietly(pstmt, res); + } + return result; + } + + @Override + public List<PartitionDescProto> getPartitionsByFilter(PartitionsByFilterProto request) { + throw new TajoRuntimeException(new UnsupportedException()); + } + + @Override + public List<PartitionDescProto> getPartitionsByAlgebra(PartitionsByAlgebraProto request) throws + UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException, + PartitionNotFoundException, UnsupportedException { + Connection conn = null; + PreparedStatement pstmt = null; + ResultSet res = null; + int currentIndex = 1; + String selectStatement = null; + + List<PartitionDescProto> partitions = TUtil.newList(); + List<PartitionFilterSet> filterSets = TUtil.newList(); + + int databaseId = getDatabaseId(request.getDatabaseName()); + int tableId = getTableId(databaseId, request.getDatabaseName(), request.getTableName()); + if (!existPartitionMethod(request.getDatabaseName(), request.getTableName())) { + throw new UndefinedPartitionMethodException(request.getTableName()); + } + + if (!existPartitionsOnCatalog(tableId)) { + throw new PartitionNotFoundException(request.getTableName()); + } + + try { + TableDescProto tableDesc = getTable(request.getDatabaseName(), request.getTableName()); + + selectStatement = getSelectStatementForPartitions(tableDesc.getTableName(), tableDesc.getPartition() + .getExpressionSchema().getFieldsList(), request.getAlgebra(), filterSets); + + conn = getConnection(); + pstmt = conn.prepareStatement(selectStatement); + + // Set table id by force because first parameter of all direct sql is table id + pstmt.setInt(currentIndex, tableId); + currentIndex++; + + for (PartitionFilterSet filter : filterSets) { + // Set table id by force because all filters have table id as first parameter. + pstmt.setInt(currentIndex, tableId); + currentIndex++; + + for (Pair<Type, Object> parameter : filter.getParameters()) { + switch (parameter.getFirst()) { + case BOOLEAN: + pstmt.setBoolean(currentIndex, (Boolean)parameter.getSecond()); + break; + case INT8: + pstmt.setLong(currentIndex, (Long) parameter.getSecond()); + break; + case FLOAT8: + pstmt.setDouble(currentIndex, (Double) parameter.getSecond()); + break; + case DATE: + pstmt.setDate(currentIndex, (Date) parameter.getSecond()); + break; + case TIMESTAMP: + pstmt.setTimestamp(currentIndex, (Timestamp) parameter.getSecond()); + break; + case TIME: + pstmt.setTime(currentIndex, (Time) parameter.getSecond()); + break; + default: + pstmt.setString(currentIndex, (String) parameter.getSecond()); + break; + } + currentIndex++; + } + } + + res = pstmt.executeQuery(); + + while (res.next()) { + PartitionDescProto.Builder builder = PartitionDescProto.newBuilder(); + + builder.setId(res.getInt(COL_PARTITIONS_PK)); + builder.setPartitionName(res.getString("PARTITION_NAME")); + builder.setPath(res.getString("PATH")); + builder.setNumBytes(res.getLong(COL_PARTITION_BYTES)); + builder.setNumFiles(res.getLong(COL_PARTITION_FILES)); + + partitions.add(builder.build()); + } + } catch (TajoException se) { + throw new TajoInternalError(se); + } catch (SQLException se) { + throw new TajoInternalError(se); + } finally { + CatalogUtil.closeQuietly(pstmt, res); + } + + return partitions; + } + + /** + * Create a select statement and parameters for querying partitions and partition keys in CatalogStore. + * + * For example, consider you have a partitioned table for three columns (i.e., col1, col2, col3). + * Assume that an user gives a condition WHERE (col1 ='1' or col1 = '100') and col3 > 20. + * There is no filter condition corresponding to col2. + * + * Then, the sql would be generated as following: + * + * SELECT A.PARTITION_ID, A.PARTITION_NAME, A.PATH FROM PARTITIONS A + * WHERE A.TID = ? + * AND A.PARTITION_ID IN ( + * SELECT T1.PARTITION_ID FROM PARTITION_KEYS T1 + * JOIN PARTITION_KEYS T2 ON T1.TID=T2.TID AND T1.PARTITION_ID = T2.PARTITION_ID AND T2.TID = ? + * AND ( T2.COLUMN_NAME = 'col2' AND T2.PARTITION_VALUE IS NOT NULL ) + * JOIN PARTITION_KEYS T3 ON T1.TID=T3.TID AND T1.PARTITION_ID = T3.PARTITION_ID AND T3.TID = ? + * AND ( T3.COLUMN_NAME = 'col3' AND T3.PARTITION_VALUE > ? ) + * WHERE T1.TID = ? AND ( T1.COLUMN_NAME = 'col1' AND T1.PARTITION_VALUE = ? ) + * OR ( T1.COLUMN_NAME = 'col1' AND T1.PARTITION_VALUE = ? ) + ) + * + * @param partitionColumns + * @param json + * @param filterSets + * @return + * @throws TajoException + * @throws SQLException + */ + private String getSelectStatementForPartitions(String tableName, List<ColumnProto> partitionColumns, String json, + List<PartitionFilterSet> filterSets) throws TajoException { — End diff – At least, you should specify that ```filterSets``` is one of the results of this method.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r39944975

          — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java —
          @@ -2228,6 +2244,257 @@ private void setPartitionKeys(int pid, PartitionDescProto.Builder partitionDesc)
          return partitions;
          }

          + /**
          + * Check if list of partitions exist on catalog.
          + *
          + *
          + * @param databaseId
          + * @param tableId
          + * @return
          + */
          + public boolean existPartitionsOnCatalog(int tableId) {
          + Connection conn = null;
          + ResultSet res = null;
          + PreparedStatement pstmt = null;
          + boolean result = false;
          +
          + try {
          + String sql = "SELECT COUNT CNT FROM "
          + + TB_PARTTIONS +" WHERE " + COL_TABLES_PK + " = ? ";
          +
          + if (LOG.isDebugEnabled())

          { + LOG.debug(sql); + }

          +
          + conn = getConnection();
          + pstmt = conn.prepareStatement(sql);
          + pstmt.setInt(1, tableId);
          + res = pstmt.executeQuery();
          +
          + if (res.next()) {
          + if (res.getInt("CNT") > 0)

          { + result = true; + }

          + }
          + } catch (SQLException se)

          { + throw new TajoInternalError(se); + } finally { + CatalogUtil.closeQuietly(pstmt, res); + }
          + return result;
          + }
          +
          + @Override
          + public List<PartitionDescProto> getPartitionsByFilter(PartitionsByFilterProto request) { + throw new TajoRuntimeException(new UnsupportedException()); + }
          +
          + @Override
          + public List<PartitionDescProto> getPartitionsByAlgebra(PartitionsByAlgebraProto request) throws
          + UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException,
          + PartitionNotFoundException, UnsupportedException {
          + Connection conn = null;
          + PreparedStatement pstmt = null;
          + ResultSet res = null;
          + int currentIndex = 1;
          + String selectStatement = null;
          +
          + List<PartitionDescProto> partitions = TUtil.newList();
          + List<PartitionFilterSet> filterSets = TUtil.newList();
          +
          + int databaseId = getDatabaseId(request.getDatabaseName());
          + int tableId = getTableId(databaseId, request.getDatabaseName(), request.getTableName());
          + if (!existPartitionMethod(request.getDatabaseName(), request.getTableName())) { + throw new UndefinedPartitionMethodException(request.getTableName()); + }
          +
          + if (!existPartitionsOnCatalog(tableId)) { + throw new PartitionNotFoundException(request.getTableName()); + }
          +
          + try {
          + TableDescProto tableDesc = getTable(request.getDatabaseName(), request.getTableName());
          +
          + selectStatement = getSelectStatementForPartitions(tableDesc.getTableName(), tableDesc.getPartition()
          + .getExpressionSchema().getFieldsList(), request.getAlgebra(), filterSets);
          +
          + conn = getConnection();
          + pstmt = conn.prepareStatement(selectStatement);
          +
          + // Set table id by force because first parameter of all direct sql is table id
          + pstmt.setInt(currentIndex, tableId);
          + currentIndex++;
          +
          + for (PartitionFilterSet filter : filterSets) {
          + // Set table id by force because all filters have table id as first parameter.
          + pstmt.setInt(currentIndex, tableId);
          + currentIndex++;
          +
          + for (Pair<Type, Object> parameter : filter.getParameters()) {
          + switch (parameter.getFirst()) { + case BOOLEAN: + pstmt.setBoolean(currentIndex, (Boolean)parameter.getSecond()); + break; + case INT8: + pstmt.setLong(currentIndex, (Long) parameter.getSecond()); + break; + case FLOAT8: + pstmt.setDouble(currentIndex, (Double) parameter.getSecond()); + break; + case DATE: + pstmt.setDate(currentIndex, (Date) parameter.getSecond()); + break; + case TIMESTAMP: + pstmt.setTimestamp(currentIndex, (Timestamp) parameter.getSecond()); + break; + case TIME: + pstmt.setTime(currentIndex, (Time) parameter.getSecond()); + break; + default: + pstmt.setString(currentIndex, (String) parameter.getSecond()); + break; + }
          + currentIndex++;
          + }
          + }
          +
          + res = pstmt.executeQuery();
          +
          + while (res.next()) { + PartitionDescProto.Builder builder = PartitionDescProto.newBuilder(); + + builder.setId(res.getInt(COL_PARTITIONS_PK)); + builder.setPartitionName(res.getString("PARTITION_NAME")); + builder.setPath(res.getString("PATH")); + builder.setNumBytes(res.getLong(COL_PARTITION_BYTES)); + builder.setNumFiles(res.getLong(COL_PARTITION_FILES)); + + partitions.add(builder.build()); + }
          + } catch (TajoException se) { + throw new TajoInternalError(se); + }

          catch (SQLException se)

          { + throw new TajoInternalError(se); + }

          finally

          { + CatalogUtil.closeQuietly(pstmt, res); + }

          +
          + return partitions;
          + }
          +
          + /**
          + * Create a select statement and parameters for querying partitions and partition keys in CatalogStore.
          + *
          + * For example, consider you have a partitioned table for three columns (i.e., col1, col2, col3).
          + * Assume that an user gives a condition WHERE (col1 ='1' or col1 = '100') and col3 > 20.
          + * There is no filter condition corresponding to col2.
          + *
          + * Then, the sql would be generated as following:
          + *
          + * SELECT A.PARTITION_ID, A.PARTITION_NAME, A.PATH FROM PARTITIONS A
          + * WHERE A.TID = ?
          + * AND A.PARTITION_ID IN (
          + * SELECT T1.PARTITION_ID FROM PARTITION_KEYS T1
          + * JOIN PARTITION_KEYS T2 ON T1.TID=T2.TID AND T1.PARTITION_ID = T2.PARTITION_ID AND T2.TID = ?
          + * AND ( T2.COLUMN_NAME = 'col2' AND T2.PARTITION_VALUE IS NOT NULL )
          + * JOIN PARTITION_KEYS T3 ON T1.TID=T3.TID AND T1.PARTITION_ID = T3.PARTITION_ID AND T3.TID = ?
          + * AND ( T3.COLUMN_NAME = 'col3' AND T3.PARTITION_VALUE > ? )
          + * WHERE T1.TID = ? AND ( T1.COLUMN_NAME = 'col1' AND T1.PARTITION_VALUE = ? )
          + * OR ( T1.COLUMN_NAME = 'col1' AND T1.PARTITION_VALUE = ? )
          — End diff –

          I cannot understand this example. Would you tell me why did you use expensive self-join on partition_keys? This definitely degrades the performance as the more partitions are stored.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r39944975 — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java — @@ -2228,6 +2244,257 @@ private void setPartitionKeys(int pid, PartitionDescProto.Builder partitionDesc) return partitions; } + /** + * Check if list of partitions exist on catalog. + * + * + * @param databaseId + * @param tableId + * @return + */ + public boolean existPartitionsOnCatalog(int tableId) { + Connection conn = null; + ResultSet res = null; + PreparedStatement pstmt = null; + boolean result = false; + + try { + String sql = "SELECT COUNT CNT FROM " + + TB_PARTTIONS +" WHERE " + COL_TABLES_PK + " = ? "; + + if (LOG.isDebugEnabled()) { + LOG.debug(sql); + } + + conn = getConnection(); + pstmt = conn.prepareStatement(sql); + pstmt.setInt(1, tableId); + res = pstmt.executeQuery(); + + if (res.next()) { + if (res.getInt("CNT") > 0) { + result = true; + } + } + } catch (SQLException se) { + throw new TajoInternalError(se); + } finally { + CatalogUtil.closeQuietly(pstmt, res); + } + return result; + } + + @Override + public List<PartitionDescProto> getPartitionsByFilter(PartitionsByFilterProto request) { + throw new TajoRuntimeException(new UnsupportedException()); + } + + @Override + public List<PartitionDescProto> getPartitionsByAlgebra(PartitionsByAlgebraProto request) throws + UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException, + PartitionNotFoundException, UnsupportedException { + Connection conn = null; + PreparedStatement pstmt = null; + ResultSet res = null; + int currentIndex = 1; + String selectStatement = null; + + List<PartitionDescProto> partitions = TUtil.newList(); + List<PartitionFilterSet> filterSets = TUtil.newList(); + + int databaseId = getDatabaseId(request.getDatabaseName()); + int tableId = getTableId(databaseId, request.getDatabaseName(), request.getTableName()); + if (!existPartitionMethod(request.getDatabaseName(), request.getTableName())) { + throw new UndefinedPartitionMethodException(request.getTableName()); + } + + if (!existPartitionsOnCatalog(tableId)) { + throw new PartitionNotFoundException(request.getTableName()); + } + + try { + TableDescProto tableDesc = getTable(request.getDatabaseName(), request.getTableName()); + + selectStatement = getSelectStatementForPartitions(tableDesc.getTableName(), tableDesc.getPartition() + .getExpressionSchema().getFieldsList(), request.getAlgebra(), filterSets); + + conn = getConnection(); + pstmt = conn.prepareStatement(selectStatement); + + // Set table id by force because first parameter of all direct sql is table id + pstmt.setInt(currentIndex, tableId); + currentIndex++; + + for (PartitionFilterSet filter : filterSets) { + // Set table id by force because all filters have table id as first parameter. + pstmt.setInt(currentIndex, tableId); + currentIndex++; + + for (Pair<Type, Object> parameter : filter.getParameters()) { + switch (parameter.getFirst()) { + case BOOLEAN: + pstmt.setBoolean(currentIndex, (Boolean)parameter.getSecond()); + break; + case INT8: + pstmt.setLong(currentIndex, (Long) parameter.getSecond()); + break; + case FLOAT8: + pstmt.setDouble(currentIndex, (Double) parameter.getSecond()); + break; + case DATE: + pstmt.setDate(currentIndex, (Date) parameter.getSecond()); + break; + case TIMESTAMP: + pstmt.setTimestamp(currentIndex, (Timestamp) parameter.getSecond()); + break; + case TIME: + pstmt.setTime(currentIndex, (Time) parameter.getSecond()); + break; + default: + pstmt.setString(currentIndex, (String) parameter.getSecond()); + break; + } + currentIndex++; + } + } + + res = pstmt.executeQuery(); + + while (res.next()) { + PartitionDescProto.Builder builder = PartitionDescProto.newBuilder(); + + builder.setId(res.getInt(COL_PARTITIONS_PK)); + builder.setPartitionName(res.getString("PARTITION_NAME")); + builder.setPath(res.getString("PATH")); + builder.setNumBytes(res.getLong(COL_PARTITION_BYTES)); + builder.setNumFiles(res.getLong(COL_PARTITION_FILES)); + + partitions.add(builder.build()); + } + } catch (TajoException se) { + throw new TajoInternalError(se); + } catch (SQLException se) { + throw new TajoInternalError(se); + } finally { + CatalogUtil.closeQuietly(pstmt, res); + } + + return partitions; + } + + /** + * Create a select statement and parameters for querying partitions and partition keys in CatalogStore. + * + * For example, consider you have a partitioned table for three columns (i.e., col1, col2, col3). + * Assume that an user gives a condition WHERE (col1 ='1' or col1 = '100') and col3 > 20. + * There is no filter condition corresponding to col2. + * + * Then, the sql would be generated as following: + * + * SELECT A.PARTITION_ID, A.PARTITION_NAME, A.PATH FROM PARTITIONS A + * WHERE A.TID = ? + * AND A.PARTITION_ID IN ( + * SELECT T1.PARTITION_ID FROM PARTITION_KEYS T1 + * JOIN PARTITION_KEYS T2 ON T1.TID=T2.TID AND T1.PARTITION_ID = T2.PARTITION_ID AND T2.TID = ? + * AND ( T2.COLUMN_NAME = 'col2' AND T2.PARTITION_VALUE IS NOT NULL ) + * JOIN PARTITION_KEYS T3 ON T1.TID=T3.TID AND T1.PARTITION_ID = T3.PARTITION_ID AND T3.TID = ? + * AND ( T3.COLUMN_NAME = 'col3' AND T3.PARTITION_VALUE > ? ) + * WHERE T1.TID = ? AND ( T1.COLUMN_NAME = 'col1' AND T1.PARTITION_VALUE = ? ) + * OR ( T1.COLUMN_NAME = 'col1' AND T1.PARTITION_VALUE = ? ) — End diff – I cannot understand this example. Would you tell me why did you use expensive self-join on partition_keys? This definitely degrades the performance as the more partitions are stored.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r39945077

          — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/CatalogStore.java —
          @@ -97,13 +97,31 @@ boolean existPartitionMethod(String databaseName, String tableName) throws Undef

          • @return
          • @throws TajoException
            */
          • List<CatalogProtos.PartitionDescProto> getPartitions(String databaseName, String tableName) throws
          • UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException;
            + List<CatalogProtos.PartitionDescProto> getAllPartitions(String databaseName, String tableName) throws
              • End diff –

          I think ```getPartitionsOfTable()``` is a more appropriate name.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r39945077 — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/CatalogStore.java — @@ -97,13 +97,31 @@ boolean existPartitionMethod(String databaseName, String tableName) throws Undef @return @throws TajoException */ List<CatalogProtos.PartitionDescProto> getPartitions(String databaseName, String tableName) throws UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException; + List<CatalogProtos.PartitionDescProto> getAllPartitions(String databaseName, String tableName) throws End diff – I think ```getPartitionsOfTable()``` is a more appropriate name.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r39945279

          — Diff: tajo-catalog/tajo-catalog-server/src/main/resources/schemas/derby/derby.xml —
          @@ -173,7 +174,9 @@
          TID INT NOT NULL REFERENCES TABLES (TID) ON DELETE CASCADE,
          PARTITION_NAME VARCHAR(767),
          PATH VARCHAR(1024),

          • CONSTRAINT C_PARTITIONS_PK PRIMARY KEY (PARTITION_ID),
            + NUM_BYTES BIGINT,
            + NUM_FILES BIGINT,
            + CONSTRAINT C_PARTITIONS_PK PRIMARY KEY (PARTITION_ID),
              • End diff –

          Please fix the indent.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r39945279 — Diff: tajo-catalog/tajo-catalog-server/src/main/resources/schemas/derby/derby.xml — @@ -173,7 +174,9 @@ TID INT NOT NULL REFERENCES TABLES (TID) ON DELETE CASCADE, PARTITION_NAME VARCHAR(767), PATH VARCHAR(1024), CONSTRAINT C_PARTITIONS_PK PRIMARY KEY (PARTITION_ID), + NUM_BYTES BIGINT, + NUM_FILES BIGINT, + CONSTRAINT C_PARTITIONS_PK PRIMARY KEY (PARTITION_ID), End diff – Please fix the indent.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r39945296

          — Diff: tajo-catalog/tajo-catalog-server/src/main/resources/schemas/mariadb/mariadb.xml —
          @@ -150,6 +151,8 @@
          TID INT NOT NULL,
          PARTITION_NAME VARCHAR(255) BINARY,
          PATH VARCHAR(4096) BINARY,
          + NUM_BYTES BIGINT,
          + NUM_FILES BIGINT,
          — End diff –

          Please fix the indent.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r39945296 — Diff: tajo-catalog/tajo-catalog-server/src/main/resources/schemas/mariadb/mariadb.xml — @@ -150,6 +151,8 @@ TID INT NOT NULL, PARTITION_NAME VARCHAR(255) BINARY, PATH VARCHAR(4096) BINARY, + NUM_BYTES BIGINT, + NUM_FILES BIGINT, — End diff – Please fix the indent.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r39945422

          — Diff: tajo-catalog/tajo-catalog-server/src/main/resources/schemas/mysql/mysql.xml —
          @@ -154,6 +155,8 @@
          TID INT NOT NULL,
          PARTITION_NAME VARCHAR(255) BINARY,
          PATH VARCHAR(4096) BINARY,
          + NUM_BYTES BIGINT,
          — End diff –

          Please fix the indent.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r39945422 — Diff: tajo-catalog/tajo-catalog-server/src/main/resources/schemas/mysql/mysql.xml — @@ -154,6 +155,8 @@ TID INT NOT NULL, PARTITION_NAME VARCHAR(255) BINARY, PATH VARCHAR(4096) BINARY, + NUM_BYTES BIGINT, — End diff – Please fix the indent.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r39945612

          — Diff: tajo-core/src/main/java/org/apache/tajo/querymaster/Query.java —
          @@ -509,6 +510,18 @@ private QueryState finalizeQuery(Query query, QueryCompletedEvent event) {
          if (queryContext.hasOutputTableUri() && queryContext.hasPartition()) {
          List<PartitionDescProto> partitions = query.getPartitions();
          if (partitions != null) {
          + // Set contents length and file count to PartitionDescProto by listing final output directories.
          + List<PartitionDescProto> finalPartitions = TUtil.newList();
          + FileSystem fileSystem = finalOutputDir.getFileSystem(query.systemConf);
          + for (PartitionDescProto partition : partitions)

          { + PartitionDescProto.Builder builder = partition.toBuilder(); + Path partitionPath = new Path(finalOutputDir, partition.getPath()); + ContentSummary contentSummary = fileSystem.getContentSummary(partitionPath); + builder.setNumBytes(contentSummary.getLength()); + builder.setNumFiles(contentSummary.getFileCount()); + finalPartitions.add(builder.build()); + }

          — End diff –

          It would be better to modularize this routine.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r39945612 — Diff: tajo-core/src/main/java/org/apache/tajo/querymaster/Query.java — @@ -509,6 +510,18 @@ private QueryState finalizeQuery(Query query, QueryCompletedEvent event) { if (queryContext.hasOutputTableUri() && queryContext.hasPartition()) { List<PartitionDescProto> partitions = query.getPartitions(); if (partitions != null) { + // Set contents length and file count to PartitionDescProto by listing final output directories. + List<PartitionDescProto> finalPartitions = TUtil.newList(); + FileSystem fileSystem = finalOutputDir.getFileSystem(query.systemConf); + for (PartitionDescProto partition : partitions) { + PartitionDescProto.Builder builder = partition.toBuilder(); + Path partitionPath = new Path(finalOutputDir, partition.getPath()); + ContentSummary contentSummary = fileSystem.getContentSummary(partitionPath); + builder.setNumBytes(contentSummary.getLength()); + builder.setNumFiles(contentSummary.getFileCount()); + finalPartitions.add(builder.build()); + } — End diff – It would be better to modularize this routine.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r39945659

          — Diff: tajo-plan/src/main/java/org/apache/tajo/plan/expr/AlgebraicUtil.java —
          @@ -493,4 +497,190 @@ public Object visitTimeLiteral(Object ctx, Stack<Expr> stack, TimeLiteral expr)
          return super.visitTimeLiteral(ctx, stack, expr);
          }
          }
          +
          + /**
          + * Find the top expr matched to type from the given expr
          + *
          + * @param expr start expr
          + * @param type to find
          + * @return a found expr
          + */
          + public static <T extends Expr> T findTopExpr(Expr expr, OpType type) throws TajoException {
          + Preconditions.checkNotNull(expr);
          + Preconditions.checkNotNull(type);
          +
          + ExprFinder finder = new ExprFinder(type);
          + finder.visit(null, new Stack<Expr>(), expr);
          +
          + if (finder.getFoundExprs().size() == 0)

          { + return null; + }

          + return (T) finder.getFoundExprs().get(0);
          + }
          +
          + private static class ExprFinder extends SimpleAlgebraVisitor<Object, Expr> {
          — End diff –

          We have already ExprFinder.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r39945659 — Diff: tajo-plan/src/main/java/org/apache/tajo/plan/expr/AlgebraicUtil.java — @@ -493,4 +497,190 @@ public Object visitTimeLiteral(Object ctx, Stack<Expr> stack, TimeLiteral expr) return super.visitTimeLiteral(ctx, stack, expr); } } + + /** + * Find the top expr matched to type from the given expr + * + * @param expr start expr + * @param type to find + * @return a found expr + */ + public static <T extends Expr> T findTopExpr(Expr expr, OpType type) throws TajoException { + Preconditions.checkNotNull(expr); + Preconditions.checkNotNull(type); + + ExprFinder finder = new ExprFinder(type); + finder.visit(null, new Stack<Expr>(), expr); + + if (finder.getFoundExprs().size() == 0) { + return null; + } + return (T) finder.getFoundExprs().get(0); + } + + private static class ExprFinder extends SimpleAlgebraVisitor<Object, Expr> { — End diff – We have already ExprFinder.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r39945793

          — Diff: tajo-plan/src/main/java/org/apache/tajo/plan/expr/AlgebraicUtil.java —
          @@ -493,4 +497,190 @@ public Object visitTimeLiteral(Object ctx, Stack<Expr> stack, TimeLiteral expr)
          return super.visitTimeLiteral(ctx, stack, expr);
          }
          }
          +
          + /**
          + * Find the top expr matched to type from the given expr
          + *
          + * @param expr start expr
          + * @param type to find
          + * @return a found expr
          + */
          + public static <T extends Expr> T findTopExpr(Expr expr, OpType type) throws TajoException {
          + Preconditions.checkNotNull(expr);
          + Preconditions.checkNotNull(type);
          +
          + ExprFinder finder = new ExprFinder(type);
          + finder.visit(null, new Stack<Expr>(), expr);
          +
          + if (finder.getFoundExprs().size() == 0)

          { + return null; + }

          + return (T) finder.getFoundExprs().get(0);
          + }
          +
          + private static class ExprFinder extends SimpleAlgebraVisitor<Object, Expr> {
          + private List<Expr> list = new ArrayList<Expr>();
          + private final OpType[] tofind;
          + private boolean topmost = false;
          + private boolean finished = false;
          +
          + public ExprFinder(OpType... type)

          { + + this.tofind = type; + }

          +
          + public ExprFinder(OpType[] type, boolean topmost)

          { + this(type); + this.topmost = topmost; + }

          +
          + @Override
          + public Expr visit(Object ctx, Stack<Expr> stack, Expr expr) throws TajoException {
          + if (!finished) {
          + for (OpType type : tofind) {
          + if (expr.getType() == type)

          { + list.add(expr); + }

          + if (topmost && list.size() > 0)

          { + finished = true; + }

          + }
          + }
          + return super.visit(ctx, stack, expr);
          + }
          +
          + public List<Expr> getFoundExprs()

          { + return list; + }

          +
          + }
          +
          + public static Expr[] toConjunctiveNormalFormArray(Expr expr) {
          — End diff –

          Unnecessary change.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r39945793 — Diff: tajo-plan/src/main/java/org/apache/tajo/plan/expr/AlgebraicUtil.java — @@ -493,4 +497,190 @@ public Object visitTimeLiteral(Object ctx, Stack<Expr> stack, TimeLiteral expr) return super.visitTimeLiteral(ctx, stack, expr); } } + + /** + * Find the top expr matched to type from the given expr + * + * @param expr start expr + * @param type to find + * @return a found expr + */ + public static <T extends Expr> T findTopExpr(Expr expr, OpType type) throws TajoException { + Preconditions.checkNotNull(expr); + Preconditions.checkNotNull(type); + + ExprFinder finder = new ExprFinder(type); + finder.visit(null, new Stack<Expr>(), expr); + + if (finder.getFoundExprs().size() == 0) { + return null; + } + return (T) finder.getFoundExprs().get(0); + } + + private static class ExprFinder extends SimpleAlgebraVisitor<Object, Expr> { + private List<Expr> list = new ArrayList<Expr>(); + private final OpType[] tofind; + private boolean topmost = false; + private boolean finished = false; + + public ExprFinder(OpType... type) { + + this.tofind = type; + } + + public ExprFinder(OpType[] type, boolean topmost) { + this(type); + this.topmost = topmost; + } + + @Override + public Expr visit(Object ctx, Stack<Expr> stack, Expr expr) throws TajoException { + if (!finished) { + for (OpType type : tofind) { + if (expr.getType() == type) { + list.add(expr); + } + if (topmost && list.size() > 0) { + finished = true; + } + } + } + return super.visit(ctx, stack, expr); + } + + public List<Expr> getFoundExprs() { + return list; + } + + } + + public static Expr[] toConjunctiveNormalFormArray(Expr expr) { — End diff – Unnecessary change.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r39945794

          — Diff: tajo-plan/src/main/java/org/apache/tajo/plan/expr/AlgebraicUtil.java —
          @@ -493,4 +497,190 @@ public Object visitTimeLiteral(Object ctx, Stack<Expr> stack, TimeLiteral expr)
          return super.visitTimeLiteral(ctx, stack, expr);
          }
          }
          +
          + /**
          + * Find the top expr matched to type from the given expr
          + *
          + * @param expr start expr
          + * @param type to find
          + * @return a found expr
          + */
          + public static <T extends Expr> T findTopExpr(Expr expr, OpType type) throws TajoException {
          + Preconditions.checkNotNull(expr);
          + Preconditions.checkNotNull(type);
          +
          + ExprFinder finder = new ExprFinder(type);
          + finder.visit(null, new Stack<Expr>(), expr);
          +
          + if (finder.getFoundExprs().size() == 0)

          { + return null; + }

          + return (T) finder.getFoundExprs().get(0);
          + }
          +
          + private static class ExprFinder extends SimpleAlgebraVisitor<Object, Expr> {
          + private List<Expr> list = new ArrayList<Expr>();
          + private final OpType[] tofind;
          + private boolean topmost = false;
          + private boolean finished = false;
          +
          + public ExprFinder(OpType... type)

          { + + this.tofind = type; + }

          +
          + public ExprFinder(OpType[] type, boolean topmost)

          { + this(type); + this.topmost = topmost; + }

          +
          + @Override
          + public Expr visit(Object ctx, Stack<Expr> stack, Expr expr) throws TajoException {
          + if (!finished) {
          + for (OpType type : tofind) {
          + if (expr.getType() == type)

          { + list.add(expr); + }

          + if (topmost && list.size() > 0)

          { + finished = true; + }

          + }
          + }
          + return super.visit(ctx, stack, expr);
          + }
          +
          + public List<Expr> getFoundExprs()

          { + return list; + }

          +
          + }
          +
          + public static Expr[] toConjunctiveNormalFormArray(Expr expr)

          { + List<Expr> list = new ArrayList<Expr>(); + toConjunctiveNormalFormArrayRecursive(expr, list); + return list.toArray(new Expr[list.size()]); + }

          +
          + private static void toConjunctiveNormalFormArrayRecursive(Expr node, List<Expr> found) {
          — End diff –

          Unnecessary change.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r39945794 — Diff: tajo-plan/src/main/java/org/apache/tajo/plan/expr/AlgebraicUtil.java — @@ -493,4 +497,190 @@ public Object visitTimeLiteral(Object ctx, Stack<Expr> stack, TimeLiteral expr) return super.visitTimeLiteral(ctx, stack, expr); } } + + /** + * Find the top expr matched to type from the given expr + * + * @param expr start expr + * @param type to find + * @return a found expr + */ + public static <T extends Expr> T findTopExpr(Expr expr, OpType type) throws TajoException { + Preconditions.checkNotNull(expr); + Preconditions.checkNotNull(type); + + ExprFinder finder = new ExprFinder(type); + finder.visit(null, new Stack<Expr>(), expr); + + if (finder.getFoundExprs().size() == 0) { + return null; + } + return (T) finder.getFoundExprs().get(0); + } + + private static class ExprFinder extends SimpleAlgebraVisitor<Object, Expr> { + private List<Expr> list = new ArrayList<Expr>(); + private final OpType[] tofind; + private boolean topmost = false; + private boolean finished = false; + + public ExprFinder(OpType... type) { + + this.tofind = type; + } + + public ExprFinder(OpType[] type, boolean topmost) { + this(type); + this.topmost = topmost; + } + + @Override + public Expr visit(Object ctx, Stack<Expr> stack, Expr expr) throws TajoException { + if (!finished) { + for (OpType type : tofind) { + if (expr.getType() == type) { + list.add(expr); + } + if (topmost && list.size() > 0) { + finished = true; + } + } + } + return super.visit(ctx, stack, expr); + } + + public List<Expr> getFoundExprs() { + return list; + } + + } + + public static Expr[] toConjunctiveNormalFormArray(Expr expr) { + List<Expr> list = new ArrayList<Expr>(); + toConjunctiveNormalFormArrayRecursive(expr, list); + return list.toArray(new Expr[list.size()]); + } + + private static void toConjunctiveNormalFormArrayRecursive(Expr node, List<Expr> found) { — End diff – Unnecessary change.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r39945983

          — Diff: tajo-plan/src/main/java/org/apache/tajo/plan/expr/AlgebraicUtil.java —
          @@ -493,4 +497,190 @@ public Object visitTimeLiteral(Object ctx, Stack<Expr> stack, TimeLiteral expr)
          return super.visitTimeLiteral(ctx, stack, expr);
          }
          }
          +
          + /**
          + * Find the top expr matched to type from the given expr
          + *
          + * @param expr start expr
          + * @param type to find
          + * @return a found expr
          + */
          + public static <T extends Expr> T findTopExpr(Expr expr, OpType type) throws TajoException {
          + Preconditions.checkNotNull(expr);
          + Preconditions.checkNotNull(type);
          +
          + ExprFinder finder = new ExprFinder(type);
          + finder.visit(null, new Stack<Expr>(), expr);
          +
          + if (finder.getFoundExprs().size() == 0)

          { + return null; + }

          + return (T) finder.getFoundExprs().get(0);
          + }
          +
          + private static class ExprFinder extends SimpleAlgebraVisitor<Object, Expr> {
          + private List<Expr> list = new ArrayList<Expr>();
          + private final OpType[] tofind;
          + private boolean topmost = false;
          + private boolean finished = false;
          +
          + public ExprFinder(OpType... type)

          { + + this.tofind = type; + }

          +
          + public ExprFinder(OpType[] type, boolean topmost)

          { + this(type); + this.topmost = topmost; + }

          +
          + @Override
          + public Expr visit(Object ctx, Stack<Expr> stack, Expr expr) throws TajoException {
          + if (!finished) {
          + for (OpType type : tofind) {
          + if (expr.getType() == type)

          { + list.add(expr); + }

          + if (topmost && list.size() > 0)

          { + finished = true; + }

          + }
          + }
          + return super.visit(ctx, stack, expr);
          + }
          +
          + public List<Expr> getFoundExprs()

          { + return list; + }

          +
          + }
          +
          + public static Expr[] toConjunctiveNormalFormArray(Expr expr)

          { + List<Expr> list = new ArrayList<Expr>(); + toConjunctiveNormalFormArrayRecursive(expr, list); + return list.toArray(new Expr[list.size()]); + }

          +
          + private static void toConjunctiveNormalFormArrayRecursive(Expr node, List<Expr> found) {
          + if (node.getType() == OpType.And)

          { + toConjunctiveNormalFormArrayRecursive(((BinaryOperator) node).getLeft(), found); + toConjunctiveNormalFormArrayRecursive(((BinaryOperator) node).getRight(), found); + }

          else

          { + found.add(node); + }

          + }
          +
          + /**
          + * It finds unique columns from a Expr.
          + */
          + public static LinkedHashSet<ColumnReferenceExpr> findUniqueColumnReferences(Expr expr) throws TajoException

          { + UniqueColumnReferenceFinder finder = new UniqueColumnReferenceFinder(); + finder.visit(null, new Stack<Expr>(), expr); + return finder.getColumnRefs(); + }

          +
          + private static class UniqueColumnReferenceFinder extends SimpleAlgebraVisitor<Object, Expr> {
          — End diff –

          Why did you create this class instead of using ExprFinder? Is there any reason?

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r39945983 — Diff: tajo-plan/src/main/java/org/apache/tajo/plan/expr/AlgebraicUtil.java — @@ -493,4 +497,190 @@ public Object visitTimeLiteral(Object ctx, Stack<Expr> stack, TimeLiteral expr) return super.visitTimeLiteral(ctx, stack, expr); } } + + /** + * Find the top expr matched to type from the given expr + * + * @param expr start expr + * @param type to find + * @return a found expr + */ + public static <T extends Expr> T findTopExpr(Expr expr, OpType type) throws TajoException { + Preconditions.checkNotNull(expr); + Preconditions.checkNotNull(type); + + ExprFinder finder = new ExprFinder(type); + finder.visit(null, new Stack<Expr>(), expr); + + if (finder.getFoundExprs().size() == 0) { + return null; + } + return (T) finder.getFoundExprs().get(0); + } + + private static class ExprFinder extends SimpleAlgebraVisitor<Object, Expr> { + private List<Expr> list = new ArrayList<Expr>(); + private final OpType[] tofind; + private boolean topmost = false; + private boolean finished = false; + + public ExprFinder(OpType... type) { + + this.tofind = type; + } + + public ExprFinder(OpType[] type, boolean topmost) { + this(type); + this.topmost = topmost; + } + + @Override + public Expr visit(Object ctx, Stack<Expr> stack, Expr expr) throws TajoException { + if (!finished) { + for (OpType type : tofind) { + if (expr.getType() == type) { + list.add(expr); + } + if (topmost && list.size() > 0) { + finished = true; + } + } + } + return super.visit(ctx, stack, expr); + } + + public List<Expr> getFoundExprs() { + return list; + } + + } + + public static Expr[] toConjunctiveNormalFormArray(Expr expr) { + List<Expr> list = new ArrayList<Expr>(); + toConjunctiveNormalFormArrayRecursive(expr, list); + return list.toArray(new Expr[list.size()]); + } + + private static void toConjunctiveNormalFormArrayRecursive(Expr node, List<Expr> found) { + if (node.getType() == OpType.And) { + toConjunctiveNormalFormArrayRecursive(((BinaryOperator) node).getLeft(), found); + toConjunctiveNormalFormArrayRecursive(((BinaryOperator) node).getRight(), found); + } else { + found.add(node); + } + } + + /** + * It finds unique columns from a Expr. + */ + public static LinkedHashSet<ColumnReferenceExpr> findUniqueColumnReferences(Expr expr) throws TajoException { + UniqueColumnReferenceFinder finder = new UniqueColumnReferenceFinder(); + finder.visit(null, new Stack<Expr>(), expr); + return finder.getColumnRefs(); + } + + private static class UniqueColumnReferenceFinder extends SimpleAlgebraVisitor<Object, Expr> { — End diff – Why did you create this class instead of using ExprFinder? Is there any reason?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r39946178

          — Diff: tajo-plan/src/main/java/org/apache/tajo/plan/expr/AlgebraicUtil.java —
          @@ -493,4 +497,190 @@ public Object visitTimeLiteral(Object ctx, Stack<Expr> stack, TimeLiteral expr)
          return super.visitTimeLiteral(ctx, stack, expr);
          }
          }
          +
          + /**
          + * Find the top expr matched to type from the given expr
          + *
          + * @param expr start expr
          + * @param type to find
          + * @return a found expr
          + */
          + public static <T extends Expr> T findTopExpr(Expr expr, OpType type) throws TajoException {
          + Preconditions.checkNotNull(expr);
          + Preconditions.checkNotNull(type);
          +
          + ExprFinder finder = new ExprFinder(type);
          + finder.visit(null, new Stack<Expr>(), expr);
          +
          + if (finder.getFoundExprs().size() == 0)

          { + return null; + }

          + return (T) finder.getFoundExprs().get(0);
          + }
          +
          + private static class ExprFinder extends SimpleAlgebraVisitor<Object, Expr> {
          + private List<Expr> list = new ArrayList<Expr>();
          + private final OpType[] tofind;
          + private boolean topmost = false;
          + private boolean finished = false;
          +
          + public ExprFinder(OpType... type)

          { + + this.tofind = type; + }

          +
          + public ExprFinder(OpType[] type, boolean topmost)

          { + this(type); + this.topmost = topmost; + }

          +
          + @Override
          + public Expr visit(Object ctx, Stack<Expr> stack, Expr expr) throws TajoException {
          + if (!finished) {
          + for (OpType type : tofind) {
          + if (expr.getType() == type)

          { + list.add(expr); + }

          + if (topmost && list.size() > 0)

          { + finished = true; + }

          + }
          + }
          + return super.visit(ctx, stack, expr);
          + }
          +
          + public List<Expr> getFoundExprs()

          { + return list; + }

          +
          + }
          +
          + public static Expr[] toConjunctiveNormalFormArray(Expr expr)

          { + List<Expr> list = new ArrayList<Expr>(); + toConjunctiveNormalFormArrayRecursive(expr, list); + return list.toArray(new Expr[list.size()]); + }

          +
          + private static void toConjunctiveNormalFormArrayRecursive(Expr node, List<Expr> found) {
          + if (node.getType() == OpType.And)

          { + toConjunctiveNormalFormArrayRecursive(((BinaryOperator) node).getLeft(), found); + toConjunctiveNormalFormArrayRecursive(((BinaryOperator) node).getRight(), found); + }

          else

          { + found.add(node); + }

          + }
          +
          + /**
          + * It finds unique columns from a Expr.
          + */
          + public static LinkedHashSet<ColumnReferenceExpr> findUniqueColumnReferences(Expr expr) throws TajoException

          { + UniqueColumnReferenceFinder finder = new UniqueColumnReferenceFinder(); + finder.visit(null, new Stack<Expr>(), expr); + return finder.getColumnRefs(); + }

          +
          + private static class UniqueColumnReferenceFinder extends SimpleAlgebraVisitor<Object, Expr> {
          + private LinkedHashSet<ColumnReferenceExpr> columnSet = Sets.newLinkedHashSet();
          + private ColumnReferenceExpr field = null;
          +
          + @Override
          + public Expr visit(Object ctx, Stack<Expr> stack, Expr expr) throws TajoException {
          + if (expr.getType() == OpType.Column)

          { + field = (ColumnReferenceExpr) expr; + columnSet.add(field); + }

          + return super.visit(ctx, stack, expr);
          + }
          +
          + public LinkedHashSet<ColumnReferenceExpr> getColumnRefs()

          { + return this.columnSet; + }

          +
          + }
          +
          + /**
          + * Build Exprs for all columns with a list of filter conditions.
          + *
          + * For example, consider you have a partitioned table for three columns (i.e., col1, col2, col3).
          + * Then, this methods will create three Exprs for (col1), (col2), (col3).
          + *
          + * Assume that an user gives a condition WHERE col1 ='A' and col3 = 'C'.
          + * There is no filter condition corresponding to col2.
          + * Then, the path filter conditions are corresponding to the followings:
          + *
          + * The first Expr: col1 = 'A'
          + * The second Expr: col2 IS NOT NULL
          + * The third Expr: col3 = 'C'
          + *
          + * 'IS NOT NULL' predicate is always true against the partition path.
          + *
          + *
          + * @param partitionColumns
          + * @param conjunctiveForms
          + * @return
          + */
          + public static Expr[] getAccumulatedFiltersByExpr(String tableName,
          + List<CatalogProtos.ColumnProto> partitionColumns, Expr[] conjunctiveForms) throws TajoException {
          + Expr[] filters = new Expr[partitionColumns.size()];
          + Column target;
          +
          + for (int i = 0; i < partitionColumns.size(); i++) {
          + List<Expr> accumulatedFilters = TUtil.newList();
          + target = new Column(partitionColumns.get);
          + ColumnReferenceExpr columnReference = new ColumnReferenceExpr(tableName, target.getSimpleName());
          +
          + if (conjunctiveForms == null)

          { + accumulatedFilters.add(new IsNullPredicate(true, columnReference)); + }

          else {
          + for (Expr expr : conjunctiveForms) {
          + if (AlgebraicUtil.findUniqueColumnReferences(expr).contains(columnReference))

          { + // Accumulate one qual per level + accumulatedFilters.add(expr); + }

          + }
          +
          + if (accumulatedFilters.size() == 0)

          { + accumulatedFilters.add(new IsNullPredicate(true, columnReference)); + }

          + }
          +
          + Expr filterPerLevel = AlgebraicUtil.createSingletonExprFromCNFByExpr(
          + accumulatedFilters.toArray(new Expr[accumulatedFilters.size()]));
          + filters[i] = filterPerLevel;
          + }
          +
          + return filters;
          + }
          +
          + public static Expr createSingletonExprFromCNFByExpr(Collection<Expr> cnfExprs) {
          — End diff –

          What is the difference between this method and createSingletonExprFromCNF()?

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r39946178 — Diff: tajo-plan/src/main/java/org/apache/tajo/plan/expr/AlgebraicUtil.java — @@ -493,4 +497,190 @@ public Object visitTimeLiteral(Object ctx, Stack<Expr> stack, TimeLiteral expr) return super.visitTimeLiteral(ctx, stack, expr); } } + + /** + * Find the top expr matched to type from the given expr + * + * @param expr start expr + * @param type to find + * @return a found expr + */ + public static <T extends Expr> T findTopExpr(Expr expr, OpType type) throws TajoException { + Preconditions.checkNotNull(expr); + Preconditions.checkNotNull(type); + + ExprFinder finder = new ExprFinder(type); + finder.visit(null, new Stack<Expr>(), expr); + + if (finder.getFoundExprs().size() == 0) { + return null; + } + return (T) finder.getFoundExprs().get(0); + } + + private static class ExprFinder extends SimpleAlgebraVisitor<Object, Expr> { + private List<Expr> list = new ArrayList<Expr>(); + private final OpType[] tofind; + private boolean topmost = false; + private boolean finished = false; + + public ExprFinder(OpType... type) { + + this.tofind = type; + } + + public ExprFinder(OpType[] type, boolean topmost) { + this(type); + this.topmost = topmost; + } + + @Override + public Expr visit(Object ctx, Stack<Expr> stack, Expr expr) throws TajoException { + if (!finished) { + for (OpType type : tofind) { + if (expr.getType() == type) { + list.add(expr); + } + if (topmost && list.size() > 0) { + finished = true; + } + } + } + return super.visit(ctx, stack, expr); + } + + public List<Expr> getFoundExprs() { + return list; + } + + } + + public static Expr[] toConjunctiveNormalFormArray(Expr expr) { + List<Expr> list = new ArrayList<Expr>(); + toConjunctiveNormalFormArrayRecursive(expr, list); + return list.toArray(new Expr[list.size()]); + } + + private static void toConjunctiveNormalFormArrayRecursive(Expr node, List<Expr> found) { + if (node.getType() == OpType.And) { + toConjunctiveNormalFormArrayRecursive(((BinaryOperator) node).getLeft(), found); + toConjunctiveNormalFormArrayRecursive(((BinaryOperator) node).getRight(), found); + } else { + found.add(node); + } + } + + /** + * It finds unique columns from a Expr. + */ + public static LinkedHashSet<ColumnReferenceExpr> findUniqueColumnReferences(Expr expr) throws TajoException { + UniqueColumnReferenceFinder finder = new UniqueColumnReferenceFinder(); + finder.visit(null, new Stack<Expr>(), expr); + return finder.getColumnRefs(); + } + + private static class UniqueColumnReferenceFinder extends SimpleAlgebraVisitor<Object, Expr> { + private LinkedHashSet<ColumnReferenceExpr> columnSet = Sets.newLinkedHashSet(); + private ColumnReferenceExpr field = null; + + @Override + public Expr visit(Object ctx, Stack<Expr> stack, Expr expr) throws TajoException { + if (expr.getType() == OpType.Column) { + field = (ColumnReferenceExpr) expr; + columnSet.add(field); + } + return super.visit(ctx, stack, expr); + } + + public LinkedHashSet<ColumnReferenceExpr> getColumnRefs() { + return this.columnSet; + } + + } + + /** + * Build Exprs for all columns with a list of filter conditions. + * + * For example, consider you have a partitioned table for three columns (i.e., col1, col2, col3). + * Then, this methods will create three Exprs for (col1), (col2), (col3). + * + * Assume that an user gives a condition WHERE col1 ='A' and col3 = 'C'. + * There is no filter condition corresponding to col2. + * Then, the path filter conditions are corresponding to the followings: + * + * The first Expr: col1 = 'A' + * The second Expr: col2 IS NOT NULL + * The third Expr: col3 = 'C' + * + * 'IS NOT NULL' predicate is always true against the partition path. + * + * + * @param partitionColumns + * @param conjunctiveForms + * @return + */ + public static Expr[] getAccumulatedFiltersByExpr(String tableName, + List<CatalogProtos.ColumnProto> partitionColumns, Expr[] conjunctiveForms) throws TajoException { + Expr[] filters = new Expr [partitionColumns.size()] ; + Column target; + + for (int i = 0; i < partitionColumns.size(); i++) { + List<Expr> accumulatedFilters = TUtil.newList(); + target = new Column(partitionColumns.get ); + ColumnReferenceExpr columnReference = new ColumnReferenceExpr(tableName, target.getSimpleName()); + + if (conjunctiveForms == null) { + accumulatedFilters.add(new IsNullPredicate(true, columnReference)); + } else { + for (Expr expr : conjunctiveForms) { + if (AlgebraicUtil.findUniqueColumnReferences(expr).contains(columnReference)) { + // Accumulate one qual per level + accumulatedFilters.add(expr); + } + } + + if (accumulatedFilters.size() == 0) { + accumulatedFilters.add(new IsNullPredicate(true, columnReference)); + } + } + + Expr filterPerLevel = AlgebraicUtil.createSingletonExprFromCNFByExpr( + accumulatedFilters.toArray(new Expr [accumulatedFilters.size()] )); + filters [i] = filterPerLevel; + } + + return filters; + } + + public static Expr createSingletonExprFromCNFByExpr(Collection<Expr> cnfExprs) { — End diff – What is the difference between this method and createSingletonExprFromCNF()?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r39946230

          — Diff: tajo-plan/src/main/java/org/apache/tajo/plan/expr/AlgebraicUtil.java —
          @@ -493,4 +497,190 @@ public Object visitTimeLiteral(Object ctx, Stack<Expr> stack, TimeLiteral expr)
          return super.visitTimeLiteral(ctx, stack, expr);
          }
          }
          +
          + /**
          + * Find the top expr matched to type from the given expr
          + *
          + * @param expr start expr
          + * @param type to find
          + * @return a found expr
          + */
          + public static <T extends Expr> T findTopExpr(Expr expr, OpType type) throws TajoException {
          + Preconditions.checkNotNull(expr);
          + Preconditions.checkNotNull(type);
          +
          + ExprFinder finder = new ExprFinder(type);
          + finder.visit(null, new Stack<Expr>(), expr);
          +
          + if (finder.getFoundExprs().size() == 0)

          { + return null; + }

          + return (T) finder.getFoundExprs().get(0);
          + }
          +
          + private static class ExprFinder extends SimpleAlgebraVisitor<Object, Expr> {
          + private List<Expr> list = new ArrayList<Expr>();
          + private final OpType[] tofind;
          + private boolean topmost = false;
          + private boolean finished = false;
          +
          + public ExprFinder(OpType... type)

          { + + this.tofind = type; + }

          +
          + public ExprFinder(OpType[] type, boolean topmost)

          { + this(type); + this.topmost = topmost; + }

          +
          + @Override
          + public Expr visit(Object ctx, Stack<Expr> stack, Expr expr) throws TajoException {
          + if (!finished) {
          + for (OpType type : tofind) {
          + if (expr.getType() == type)

          { + list.add(expr); + }

          + if (topmost && list.size() > 0)

          { + finished = true; + }

          + }
          + }
          + return super.visit(ctx, stack, expr);
          + }
          +
          + public List<Expr> getFoundExprs()

          { + return list; + }

          +
          + }
          +
          + public static Expr[] toConjunctiveNormalFormArray(Expr expr)

          { + List<Expr> list = new ArrayList<Expr>(); + toConjunctiveNormalFormArrayRecursive(expr, list); + return list.toArray(new Expr[list.size()]); + }

          +
          + private static void toConjunctiveNormalFormArrayRecursive(Expr node, List<Expr> found) {
          + if (node.getType() == OpType.And)

          { + toConjunctiveNormalFormArrayRecursive(((BinaryOperator) node).getLeft(), found); + toConjunctiveNormalFormArrayRecursive(((BinaryOperator) node).getRight(), found); + }

          else

          { + found.add(node); + }

          + }
          +
          + /**
          + * It finds unique columns from a Expr.
          + */
          + public static LinkedHashSet<ColumnReferenceExpr> findUniqueColumnReferences(Expr expr) throws TajoException

          { + UniqueColumnReferenceFinder finder = new UniqueColumnReferenceFinder(); + finder.visit(null, new Stack<Expr>(), expr); + return finder.getColumnRefs(); + }

          +
          + private static class UniqueColumnReferenceFinder extends SimpleAlgebraVisitor<Object, Expr> {
          + private LinkedHashSet<ColumnReferenceExpr> columnSet = Sets.newLinkedHashSet();
          + private ColumnReferenceExpr field = null;
          +
          + @Override
          + public Expr visit(Object ctx, Stack<Expr> stack, Expr expr) throws TajoException {
          + if (expr.getType() == OpType.Column)

          { + field = (ColumnReferenceExpr) expr; + columnSet.add(field); + }

          + return super.visit(ctx, stack, expr);
          + }
          +
          + public LinkedHashSet<ColumnReferenceExpr> getColumnRefs()

          { + return this.columnSet; + }

          +
          + }
          +
          + /**
          + * Build Exprs for all columns with a list of filter conditions.
          + *
          + * For example, consider you have a partitioned table for three columns (i.e., col1, col2, col3).
          + * Then, this methods will create three Exprs for (col1), (col2), (col3).
          + *
          + * Assume that an user gives a condition WHERE col1 ='A' and col3 = 'C'.
          + * There is no filter condition corresponding to col2.
          + * Then, the path filter conditions are corresponding to the followings:
          + *
          + * The first Expr: col1 = 'A'
          + * The second Expr: col2 IS NOT NULL
          + * The third Expr: col3 = 'C'
          + *
          + * 'IS NOT NULL' predicate is always true against the partition path.
          + *
          + *
          + * @param partitionColumns
          + * @param conjunctiveForms
          + * @return
          + */
          + public static Expr[] getAccumulatedFiltersByExpr(String tableName,
          — End diff –

          Method name is not intuitive. This method just rearrange the order of cnfs.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r39946230 — Diff: tajo-plan/src/main/java/org/apache/tajo/plan/expr/AlgebraicUtil.java — @@ -493,4 +497,190 @@ public Object visitTimeLiteral(Object ctx, Stack<Expr> stack, TimeLiteral expr) return super.visitTimeLiteral(ctx, stack, expr); } } + + /** + * Find the top expr matched to type from the given expr + * + * @param expr start expr + * @param type to find + * @return a found expr + */ + public static <T extends Expr> T findTopExpr(Expr expr, OpType type) throws TajoException { + Preconditions.checkNotNull(expr); + Preconditions.checkNotNull(type); + + ExprFinder finder = new ExprFinder(type); + finder.visit(null, new Stack<Expr>(), expr); + + if (finder.getFoundExprs().size() == 0) { + return null; + } + return (T) finder.getFoundExprs().get(0); + } + + private static class ExprFinder extends SimpleAlgebraVisitor<Object, Expr> { + private List<Expr> list = new ArrayList<Expr>(); + private final OpType[] tofind; + private boolean topmost = false; + private boolean finished = false; + + public ExprFinder(OpType... type) { + + this.tofind = type; + } + + public ExprFinder(OpType[] type, boolean topmost) { + this(type); + this.topmost = topmost; + } + + @Override + public Expr visit(Object ctx, Stack<Expr> stack, Expr expr) throws TajoException { + if (!finished) { + for (OpType type : tofind) { + if (expr.getType() == type) { + list.add(expr); + } + if (topmost && list.size() > 0) { + finished = true; + } + } + } + return super.visit(ctx, stack, expr); + } + + public List<Expr> getFoundExprs() { + return list; + } + + } + + public static Expr[] toConjunctiveNormalFormArray(Expr expr) { + List<Expr> list = new ArrayList<Expr>(); + toConjunctiveNormalFormArrayRecursive(expr, list); + return list.toArray(new Expr[list.size()]); + } + + private static void toConjunctiveNormalFormArrayRecursive(Expr node, List<Expr> found) { + if (node.getType() == OpType.And) { + toConjunctiveNormalFormArrayRecursive(((BinaryOperator) node).getLeft(), found); + toConjunctiveNormalFormArrayRecursive(((BinaryOperator) node).getRight(), found); + } else { + found.add(node); + } + } + + /** + * It finds unique columns from a Expr. + */ + public static LinkedHashSet<ColumnReferenceExpr> findUniqueColumnReferences(Expr expr) throws TajoException { + UniqueColumnReferenceFinder finder = new UniqueColumnReferenceFinder(); + finder.visit(null, new Stack<Expr>(), expr); + return finder.getColumnRefs(); + } + + private static class UniqueColumnReferenceFinder extends SimpleAlgebraVisitor<Object, Expr> { + private LinkedHashSet<ColumnReferenceExpr> columnSet = Sets.newLinkedHashSet(); + private ColumnReferenceExpr field = null; + + @Override + public Expr visit(Object ctx, Stack<Expr> stack, Expr expr) throws TajoException { + if (expr.getType() == OpType.Column) { + field = (ColumnReferenceExpr) expr; + columnSet.add(field); + } + return super.visit(ctx, stack, expr); + } + + public LinkedHashSet<ColumnReferenceExpr> getColumnRefs() { + return this.columnSet; + } + + } + + /** + * Build Exprs for all columns with a list of filter conditions. + * + * For example, consider you have a partitioned table for three columns (i.e., col1, col2, col3). + * Then, this methods will create three Exprs for (col1), (col2), (col3). + * + * Assume that an user gives a condition WHERE col1 ='A' and col3 = 'C'. + * There is no filter condition corresponding to col2. + * Then, the path filter conditions are corresponding to the followings: + * + * The first Expr: col1 = 'A' + * The second Expr: col2 IS NOT NULL + * The third Expr: col3 = 'C' + * + * 'IS NOT NULL' predicate is always true against the partition path. + * + * + * @param partitionColumns + * @param conjunctiveForms + * @return + */ + public static Expr[] getAccumulatedFiltersByExpr(String tableName, — End diff – Method name is not intuitive. This method just rearrange the order of cnfs.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r39946554

          — Diff: tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/FileTablespace.java —
          @@ -315,15 +315,21 @@ public boolean accept(Path path) {
          }
          }

          + protected List<FileStatus> listStatus(Path... dirs) throws IOException

          { + return listStatus(false, dirs); + }

          +
          /**

          • List input directories.
          • Subclasses may override to, e.g., select only files matching a regular
          • expression.
            *
            + * @param check if target table is a partitioned table
            + * @param input directories
          • @return array of FileStatus objects
          • @throws IOException if zero items.
            */
          • protected List<FileStatus> listStatus(Path... dirs) throws IOException {
            + protected List<FileStatus> listStatus(boolean hasPartition, Path... dirs) throws IOException {
              • End diff –

          This method is declared as ```protected``` and not used only in ```getSplits()``` method. So, I think that we don't have to maintain two separate methods according to the ```hasPartition``` flag.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r39946554 — Diff: tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/FileTablespace.java — @@ -315,15 +315,21 @@ public boolean accept(Path path) { } } + protected List<FileStatus> listStatus(Path... dirs) throws IOException { + return listStatus(false, dirs); + } + /** List input directories. Subclasses may override to, e.g., select only files matching a regular expression. * + * @param check if target table is a partitioned table + * @param input directories @return array of FileStatus objects @throws IOException if zero items. */ protected List<FileStatus> listStatus(Path... dirs) throws IOException { + protected List<FileStatus> listStatus(boolean hasPartition, Path... dirs) throws IOException { End diff – This method is declared as ```protected``` and not used only in ```getSplits()``` method. So, I think that we don't have to maintain two separate methods according to the ```hasPartition``` flag.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r39946733

          — Diff: tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogService.java —
          @@ -172,7 +172,15 @@ PartitionDescProto getPartition(String databaseName, String tableName, String pa
          throws UndefinedPartitionException, UndefinedPartitionMethodException, UndefinedDatabaseException,
          UndefinedTableException;

          • List<PartitionDescProto> getPartitions(String databaseName, String tableName);
            + List<PartitionDescProto> getAllPartitions(String databaseName, String tableName) throws UndefinedDatabaseException,
              • End diff –

          IMO, this method must not throw PartitionNotFoundException, but return an empty list.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r39946733 — Diff: tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogService.java — @@ -172,7 +172,15 @@ PartitionDescProto getPartition(String databaseName, String tableName, String pa throws UndefinedPartitionException, UndefinedPartitionMethodException, UndefinedDatabaseException, UndefinedTableException; List<PartitionDescProto> getPartitions(String databaseName, String tableName); + List<PartitionDescProto> getAllPartitions(String databaseName, String tableName) throws UndefinedDatabaseException, End diff – IMO, this method must not throw PartitionNotFoundException, but return an empty list.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r39946736

          — Diff: tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogService.java —
          @@ -172,7 +172,15 @@ PartitionDescProto getPartition(String databaseName, String tableName, String pa
          throws UndefinedPartitionException, UndefinedPartitionMethodException, UndefinedDatabaseException,
          UndefinedTableException;

          • List<PartitionDescProto> getPartitions(String databaseName, String tableName);
            + List<PartitionDescProto> getAllPartitions(String databaseName, String tableName) throws UndefinedDatabaseException,
            + UndefinedTableException, UndefinedPartitionMethodException, PartitionNotFoundException;
            +
            + List<PartitionDescProto> getPartitionsByAlgebra(PartitionsByAlgebraProto request) throws
              • End diff –

          IMO, this method must not throw PartitionNotFoundException, but return an empty list.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r39946736 — Diff: tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogService.java — @@ -172,7 +172,15 @@ PartitionDescProto getPartition(String databaseName, String tableName, String pa throws UndefinedPartitionException, UndefinedPartitionMethodException, UndefinedDatabaseException, UndefinedTableException; List<PartitionDescProto> getPartitions(String databaseName, String tableName); + List<PartitionDescProto> getAllPartitions(String databaseName, String tableName) throws UndefinedDatabaseException, + UndefinedTableException, UndefinedPartitionMethodException, PartitionNotFoundException; + + List<PartitionDescProto> getPartitionsByAlgebra(PartitionsByAlgebraProto request) throws End diff – IMO, this method must not throw PartitionNotFoundException, but return an empty list.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r39946738

          — Diff: tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogService.java —
          @@ -172,7 +172,15 @@ PartitionDescProto getPartition(String databaseName, String tableName, String pa
          throws UndefinedPartitionException, UndefinedPartitionMethodException, UndefinedDatabaseException,
          UndefinedTableException;

          • List<PartitionDescProto> getPartitions(String databaseName, String tableName);
            + List<PartitionDescProto> getAllPartitions(String databaseName, String tableName) throws UndefinedDatabaseException,
            + UndefinedTableException, UndefinedPartitionMethodException, PartitionNotFoundException;
            +
            + List<PartitionDescProto> getPartitionsByAlgebra(PartitionsByAlgebraProto request) throws
            + UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException,
            + PartitionNotFoundException, UnsupportedException;
            +
            + List<PartitionDescProto> getPartitionsByFilter(PartitionsByFilterProto request) throws
              • End diff –

          IMO, this method must not throw PartitionNotFoundException, but return an empty list.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r39946738 — Diff: tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogService.java — @@ -172,7 +172,15 @@ PartitionDescProto getPartition(String databaseName, String tableName, String pa throws UndefinedPartitionException, UndefinedPartitionMethodException, UndefinedDatabaseException, UndefinedTableException; List<PartitionDescProto> getPartitions(String databaseName, String tableName); + List<PartitionDescProto> getAllPartitions(String databaseName, String tableName) throws UndefinedDatabaseException, + UndefinedTableException, UndefinedPartitionMethodException, PartitionNotFoundException; + + List<PartitionDescProto> getPartitionsByAlgebra(PartitionsByAlgebraProto request) throws + UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException, + PartitionNotFoundException, UnsupportedException; + + List<PartitionDescProto> getPartitionsByFilter(PartitionsByFilterProto request) throws End diff – IMO, this method must not throw PartitionNotFoundException, but return an empty list.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user blrunner commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r39947150

          — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java —
          @@ -2147,6 +2154,8 @@ private void ensurePartitionTable(String tbName, int tableId)
          builder.setId(res.getInt(COL_PARTITIONS_PK));
          builder.setPath(res.getString("PATH"));
          builder.setPartitionName(partitionName);
          + builder.setNumBytes(res.getLong(COL_PARTITION_BYTES));
          + builder.setNumFiles(res.getLong(COL_PARTITION_FILES));
          — End diff –

          I'll remove it.

          Show
          githubbot ASF GitHub Bot added a comment - Github user blrunner commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r39947150 — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java — @@ -2147,6 +2154,8 @@ private void ensurePartitionTable(String tbName, int tableId) builder.setId(res.getInt(COL_PARTITIONS_PK)); builder.setPath(res.getString("PATH")); builder.setPartitionName(partitionName); + builder.setNumBytes(res.getLong(COL_PARTITION_BYTES)); + builder.setNumFiles(res.getLong(COL_PARTITION_FILES)); — End diff – I'll remove it.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user blrunner commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r39950510

          — Diff: tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/FileTablespace.java —
          @@ -315,15 +315,21 @@ public boolean accept(Path path) {
          }
          }

          + protected List<FileStatus> listStatus(Path... dirs) throws IOException

          { + return listStatus(false, dirs); + }

          +
          /**

          • List input directories.
          • Subclasses may override to, e.g., select only files matching a regular
          • expression.
            *
            + * @param check if target table is a partitioned table
            + * @param input directories
          • @return array of FileStatus objects
          • @throws IOException if zero items.
            */
          • protected List<FileStatus> listStatus(Path... dirs) throws IOException {
            + protected List<FileStatus> listStatus(boolean hasPartition, Path... dirs) throws IOException {
              • End diff –

          I agree with your point. I've just removed ```getSplits()```.

          Show
          githubbot ASF GitHub Bot added a comment - Github user blrunner commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r39950510 — Diff: tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/FileTablespace.java — @@ -315,15 +315,21 @@ public boolean accept(Path path) { } } + protected List<FileStatus> listStatus(Path... dirs) throws IOException { + return listStatus(false, dirs); + } + /** List input directories. Subclasses may override to, e.g., select only files matching a regular expression. * + * @param check if target table is a partitioned table + * @param input directories @return array of FileStatus objects @throws IOException if zero items. */ protected List<FileStatus> listStatus(Path... dirs) throws IOException { + protected List<FileStatus> listStatus(boolean hasPartition, Path... dirs) throws IOException { End diff – I agree with your point. I've just removed ```getSplits()```.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user blrunner commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r39951872

          — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java —
          @@ -2228,6 +2244,257 @@ private void setPartitionKeys(int pid, PartitionDescProto.Builder partitionDesc)
          return partitions;
          }

          + /**
          + * Check if list of partitions exist on catalog.
          + *
          + *
          + * @param databaseId
          + * @param tableId
          + * @return
          + */
          + public boolean existPartitionsOnCatalog(int tableId) {
          + Connection conn = null;
          + ResultSet res = null;
          + PreparedStatement pstmt = null;
          + boolean result = false;
          +
          + try {
          + String sql = "SELECT COUNT CNT FROM "
          + + TB_PARTTIONS +" WHERE " + COL_TABLES_PK + " = ? ";
          +
          + if (LOG.isDebugEnabled())

          { + LOG.debug(sql); + }

          +
          + conn = getConnection();
          + pstmt = conn.prepareStatement(sql);
          + pstmt.setInt(1, tableId);
          + res = pstmt.executeQuery();
          +
          + if (res.next()) {
          + if (res.getInt("CNT") > 0)

          { + result = true; + }

          + }
          + } catch (SQLException se)

          { + throw new TajoInternalError(se); + }

          finally

          { + CatalogUtil.closeQuietly(pstmt, res); + }

          + return result;
          + }
          +
          + @Override
          + public List<PartitionDescProto> getPartitionsByFilter(PartitionsByFilterProto request) {
          + throw new TajoRuntimeException(new UnsupportedException());
          — End diff –

          If we want to provide ```getPartitionsByFilter()``` by calling ```getPartitionsByAlgebra()```, we need to convert filter conditions to algebra expressions. I need more time to implement this feature and I wish to implement it at another issue.

          Show
          githubbot ASF GitHub Bot added a comment - Github user blrunner commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r39951872 — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java — @@ -2228,6 +2244,257 @@ private void setPartitionKeys(int pid, PartitionDescProto.Builder partitionDesc) return partitions; } + /** + * Check if list of partitions exist on catalog. + * + * + * @param databaseId + * @param tableId + * @return + */ + public boolean existPartitionsOnCatalog(int tableId) { + Connection conn = null; + ResultSet res = null; + PreparedStatement pstmt = null; + boolean result = false; + + try { + String sql = "SELECT COUNT CNT FROM " + + TB_PARTTIONS +" WHERE " + COL_TABLES_PK + " = ? "; + + if (LOG.isDebugEnabled()) { + LOG.debug(sql); + } + + conn = getConnection(); + pstmt = conn.prepareStatement(sql); + pstmt.setInt(1, tableId); + res = pstmt.executeQuery(); + + if (res.next()) { + if (res.getInt("CNT") > 0) { + result = true; + } + } + } catch (SQLException se) { + throw new TajoInternalError(se); + } finally { + CatalogUtil.closeQuietly(pstmt, res); + } + return result; + } + + @Override + public List<PartitionDescProto> getPartitionsByFilter(PartitionsByFilterProto request) { + throw new TajoRuntimeException(new UnsupportedException()); — End diff – If we want to provide ```getPartitionsByFilter()``` by calling ```getPartitionsByAlgebra()```, we need to convert filter conditions to algebra expressions. I need more time to implement this feature and I wish to implement it at another issue.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user blrunner commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r40049491

          — Diff: tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogStore.java —
          @@ -845,11 +856,155 @@ public boolean existPartitionMethod(String databaseName, String tableName) throw
          }

          @Override

          • public List<CatalogProtos.PartitionDescProto> getPartitions(String databaseName,
          • String tableName) {
          • throw new UnsupportedOperationException();
            + public List<CatalogProtos.PartitionDescProto> getAllPartitions(String databaseName, String tableName)
            + throws UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException,
            + PartitionNotFoundException
            Unknown macro: { + PartitionsByFilterProto.Builder request = PartitionsByFilterProto.newBuilder(); + request.setDatabaseName(databaseName); + request.setTableName(tableName); + request.setFilter(""); + + List<PartitionDescProto> partitions = getPartitionsByFilter(request.build()); + if (partitions.size() == 0) { + throw new PartitionNotFoundException(tableName); + } else { + return partitions; + } + }

            +
            + @Override
            + public List<PartitionDescProto> getPartitionsByAlgebra(PartitionsByAlgebraProto request) throws
            + UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException,
            + PartitionNotFoundException, UnsupportedException

            Unknown macro: { + + List<PartitionDescProto> list = null; + + try { + String databaseName = request.getDatabaseName(); + String tableName = request.getTableName(); + + TableDescProto tableDesc = getTable(databaseName, tableName); + String filter = getFilter(databaseName, tableName, tableDesc.getPartition().getExpressionSchema().getFieldsList() + , request.getAlgebra()); + list = getPartitionsByFilterFromHiveMetaStore(databaseName, tableName, filter); + } catch (UnsupportedException ue) { + throw ue; + } catch (Exception se) { + throw new TajoInternalError(se); + } + + return list; + }

            +
            + private String getFilter(String databaseName, String tableName, List<ColumnProto> partitionColumns
            + , String json) throws TajoException {
            +
            + Expr[] exprs = null;
            +
            + if (json != null && !json.isEmpty())

            { + Expr algebra = JsonHelper.fromJson(json, Expr.class); + exprs = AlgebraicUtil.toConjunctiveNormalFormArray(algebra); + }

            +
            + PartitionFilterAlgebraVisitor visitor = new PartitionFilterAlgebraVisitor();
            + visitor.setIsHiveCatalog(true);
            +
            + Expr[] filters = AlgebraicUtil.getAccumulatedFiltersByExpr(databaseName + "." + tableName, partitionColumns, exprs);
            +
            + StringBuffer sb = new StringBuffer();
            +
            + // Write join clause from second column to last column.
            + Column target;
            +
            + int addedFilter = 0;
            + String result;
            + for (int i = 0; i < partitionColumns.size(); i++) {
            + target = new Column(partitionColumns.get);
            +
            + if (!(filters[i] instanceof IsNullPredicate)) {
            + visitor.setColumn(target);
            + visitor.visit(null, new Stack<Expr>(), filters[i]);
            + result = visitor.getResult();
            +
            + // If visitor build filter successfully, add filter to be used for executing hive api.
            + if (result.length() > 0) {

              • End diff –

          You're right. We need to handle above situation.

          Show
          githubbot ASF GitHub Bot added a comment - Github user blrunner commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r40049491 — Diff: tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogStore.java — @@ -845,11 +856,155 @@ public boolean existPartitionMethod(String databaseName, String tableName) throw } @Override public List<CatalogProtos.PartitionDescProto> getPartitions(String databaseName, String tableName) { throw new UnsupportedOperationException(); + public List<CatalogProtos.PartitionDescProto> getAllPartitions(String databaseName, String tableName) + throws UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException, + PartitionNotFoundException Unknown macro: { + PartitionsByFilterProto.Builder request = PartitionsByFilterProto.newBuilder(); + request.setDatabaseName(databaseName); + request.setTableName(tableName); + request.setFilter(""); + + List<PartitionDescProto> partitions = getPartitionsByFilter(request.build()); + if (partitions.size() == 0) { + throw new PartitionNotFoundException(tableName); + } else { + return partitions; + } + } + + @Override + public List<PartitionDescProto> getPartitionsByAlgebra(PartitionsByAlgebraProto request) throws + UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException, + PartitionNotFoundException, UnsupportedException Unknown macro: { + + List<PartitionDescProto> list = null; + + try { + String databaseName = request.getDatabaseName(); + String tableName = request.getTableName(); + + TableDescProto tableDesc = getTable(databaseName, tableName); + String filter = getFilter(databaseName, tableName, tableDesc.getPartition().getExpressionSchema().getFieldsList() + , request.getAlgebra()); + list = getPartitionsByFilterFromHiveMetaStore(databaseName, tableName, filter); + } catch (UnsupportedException ue) { + throw ue; + } catch (Exception se) { + throw new TajoInternalError(se); + } + + return list; + } + + private String getFilter(String databaseName, String tableName, List<ColumnProto> partitionColumns + , String json) throws TajoException { + + Expr[] exprs = null; + + if (json != null && !json.isEmpty()) { + Expr algebra = JsonHelper.fromJson(json, Expr.class); + exprs = AlgebraicUtil.toConjunctiveNormalFormArray(algebra); + } + + PartitionFilterAlgebraVisitor visitor = new PartitionFilterAlgebraVisitor(); + visitor.setIsHiveCatalog(true); + + Expr[] filters = AlgebraicUtil.getAccumulatedFiltersByExpr(databaseName + "." + tableName, partitionColumns, exprs); + + StringBuffer sb = new StringBuffer(); + + // Write join clause from second column to last column. + Column target; + + int addedFilter = 0; + String result; + for (int i = 0; i < partitionColumns.size(); i++) { + target = new Column(partitionColumns.get ); + + if (!(filters [i] instanceof IsNullPredicate)) { + visitor.setColumn(target); + visitor.visit(null, new Stack<Expr>(), filters [i] ); + result = visitor.getResult(); + + // If visitor build filter successfully, add filter to be used for executing hive api. + if (result.length() > 0) { End diff – You're right. We need to handle above situation.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user blrunner commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r40056204

          — Diff: tajo-plan/src/main/java/org/apache/tajo/plan/expr/AlgebraicUtil.java —
          @@ -493,4 +497,190 @@ public Object visitTimeLiteral(Object ctx, Stack<Expr> stack, TimeLiteral expr)
          return super.visitTimeLiteral(ctx, stack, expr);
          }
          }
          +
          + /**
          + * Find the top expr matched to type from the given expr
          + *
          + * @param expr start expr
          + * @param type to find
          + * @return a found expr
          + */
          + public static <T extends Expr> T findTopExpr(Expr expr, OpType type) throws TajoException {
          + Preconditions.checkNotNull(expr);
          + Preconditions.checkNotNull(type);
          +
          + ExprFinder finder = new ExprFinder(type);
          + finder.visit(null, new Stack<Expr>(), expr);
          +
          + if (finder.getFoundExprs().size() == 0)

          { + return null; + }

          + return (T) finder.getFoundExprs().get(0);
          + }
          +
          + private static class ExprFinder extends SimpleAlgebraVisitor<Object, Expr> {
          + private List<Expr> list = new ArrayList<Expr>();
          + private final OpType[] tofind;
          + private boolean topmost = false;
          + private boolean finished = false;
          +
          + public ExprFinder(OpType... type)

          { + + this.tofind = type; + }

          +
          + public ExprFinder(OpType[] type, boolean topmost)

          { + this(type); + this.topmost = topmost; + }

          +
          + @Override
          + public Expr visit(Object ctx, Stack<Expr> stack, Expr expr) throws TajoException {
          + if (!finished) {
          + for (OpType type : tofind) {
          + if (expr.getType() == type)

          { + list.add(expr); + }

          + if (topmost && list.size() > 0)

          { + finished = true; + }

          + }
          + }
          + return super.visit(ctx, stack, expr);
          + }
          +
          + public List<Expr> getFoundExprs()

          { + return list; + }

          +
          + }
          +
          + public static Expr[] toConjunctiveNormalFormArray(Expr expr)

          { + List<Expr> list = new ArrayList<Expr>(); + toConjunctiveNormalFormArrayRecursive(expr, list); + return list.toArray(new Expr[list.size()]); + }

          +
          + private static void toConjunctiveNormalFormArrayRecursive(Expr node, List<Expr> found) {
          — End diff –

          The method is different from existing similar method because of input parameter. I added more description.

          Show
          githubbot ASF GitHub Bot added a comment - Github user blrunner commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r40056204 — Diff: tajo-plan/src/main/java/org/apache/tajo/plan/expr/AlgebraicUtil.java — @@ -493,4 +497,190 @@ public Object visitTimeLiteral(Object ctx, Stack<Expr> stack, TimeLiteral expr) return super.visitTimeLiteral(ctx, stack, expr); } } + + /** + * Find the top expr matched to type from the given expr + * + * @param expr start expr + * @param type to find + * @return a found expr + */ + public static <T extends Expr> T findTopExpr(Expr expr, OpType type) throws TajoException { + Preconditions.checkNotNull(expr); + Preconditions.checkNotNull(type); + + ExprFinder finder = new ExprFinder(type); + finder.visit(null, new Stack<Expr>(), expr); + + if (finder.getFoundExprs().size() == 0) { + return null; + } + return (T) finder.getFoundExprs().get(0); + } + + private static class ExprFinder extends SimpleAlgebraVisitor<Object, Expr> { + private List<Expr> list = new ArrayList<Expr>(); + private final OpType[] tofind; + private boolean topmost = false; + private boolean finished = false; + + public ExprFinder(OpType... type) { + + this.tofind = type; + } + + public ExprFinder(OpType[] type, boolean topmost) { + this(type); + this.topmost = topmost; + } + + @Override + public Expr visit(Object ctx, Stack<Expr> stack, Expr expr) throws TajoException { + if (!finished) { + for (OpType type : tofind) { + if (expr.getType() == type) { + list.add(expr); + } + if (topmost && list.size() > 0) { + finished = true; + } + } + } + return super.visit(ctx, stack, expr); + } + + public List<Expr> getFoundExprs() { + return list; + } + + } + + public static Expr[] toConjunctiveNormalFormArray(Expr expr) { + List<Expr> list = new ArrayList<Expr>(); + toConjunctiveNormalFormArrayRecursive(expr, list); + return list.toArray(new Expr[list.size()]); + } + + private static void toConjunctiveNormalFormArrayRecursive(Expr node, List<Expr> found) { — End diff – The method is different from existing similar method because of input parameter. I added more description.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user blrunner commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r40060940

          — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java —
          @@ -2228,6 +2244,257 @@ private void setPartitionKeys(int pid, PartitionDescProto.Builder partitionDesc)
          return partitions;
          }

          + /**
          + * Check if list of partitions exist on catalog.
          + *
          + *
          + * @param databaseId
          + * @param tableId
          + * @return
          + */
          + public boolean existPartitionsOnCatalog(int tableId) {
          + Connection conn = null;
          + ResultSet res = null;
          + PreparedStatement pstmt = null;
          + boolean result = false;
          +
          + try {
          + String sql = "SELECT COUNT CNT FROM "
          + + TB_PARTTIONS +" WHERE " + COL_TABLES_PK + " = ? ";
          +
          + if (LOG.isDebugEnabled())

          { + LOG.debug(sql); + }

          +
          + conn = getConnection();
          + pstmt = conn.prepareStatement(sql);
          + pstmt.setInt(1, tableId);
          + res = pstmt.executeQuery();
          +
          + if (res.next()) {
          + if (res.getInt("CNT") > 0)

          { + result = true; + }

          + }
          + } catch (SQLException se)

          { + throw new TajoInternalError(se); + } finally { + CatalogUtil.closeQuietly(pstmt, res); + }
          + return result;
          + }
          +
          + @Override
          + public List<PartitionDescProto> getPartitionsByFilter(PartitionsByFilterProto request) { + throw new TajoRuntimeException(new UnsupportedException()); + }
          +
          + @Override
          + public List<PartitionDescProto> getPartitionsByAlgebra(PartitionsByAlgebraProto request) throws
          + UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException,
          + PartitionNotFoundException, UnsupportedException {
          + Connection conn = null;
          + PreparedStatement pstmt = null;
          + ResultSet res = null;
          + int currentIndex = 1;
          + String selectStatement = null;
          +
          + List<PartitionDescProto> partitions = TUtil.newList();
          + List<PartitionFilterSet> filterSets = TUtil.newList();
          +
          + int databaseId = getDatabaseId(request.getDatabaseName());
          + int tableId = getTableId(databaseId, request.getDatabaseName(), request.getTableName());
          + if (!existPartitionMethod(request.getDatabaseName(), request.getTableName())) { + throw new UndefinedPartitionMethodException(request.getTableName()); + }
          +
          + if (!existPartitionsOnCatalog(tableId)) { + throw new PartitionNotFoundException(request.getTableName()); + }
          +
          + try {
          + TableDescProto tableDesc = getTable(request.getDatabaseName(), request.getTableName());
          +
          + selectStatement = getSelectStatementForPartitions(tableDesc.getTableName(), tableDesc.getPartition()
          + .getExpressionSchema().getFieldsList(), request.getAlgebra(), filterSets);
          +
          + conn = getConnection();
          + pstmt = conn.prepareStatement(selectStatement);
          +
          + // Set table id by force because first parameter of all direct sql is table id
          + pstmt.setInt(currentIndex, tableId);
          + currentIndex++;
          +
          + for (PartitionFilterSet filter : filterSets) {
          + // Set table id by force because all filters have table id as first parameter.
          + pstmt.setInt(currentIndex, tableId);
          + currentIndex++;
          +
          + for (Pair<Type, Object> parameter : filter.getParameters()) {
          + switch (parameter.getFirst()) { + case BOOLEAN: + pstmt.setBoolean(currentIndex, (Boolean)parameter.getSecond()); + break; + case INT8: + pstmt.setLong(currentIndex, (Long) parameter.getSecond()); + break; + case FLOAT8: + pstmt.setDouble(currentIndex, (Double) parameter.getSecond()); + break; + case DATE: + pstmt.setDate(currentIndex, (Date) parameter.getSecond()); + break; + case TIMESTAMP: + pstmt.setTimestamp(currentIndex, (Timestamp) parameter.getSecond()); + break; + case TIME: + pstmt.setTime(currentIndex, (Time) parameter.getSecond()); + break; + default: + pstmt.setString(currentIndex, (String) parameter.getSecond()); + break; + }
          + currentIndex++;
          + }
          + }
          +
          + res = pstmt.executeQuery();
          +
          + while (res.next()) { + PartitionDescProto.Builder builder = PartitionDescProto.newBuilder(); + + builder.setId(res.getInt(COL_PARTITIONS_PK)); + builder.setPartitionName(res.getString("PARTITION_NAME")); + builder.setPath(res.getString("PATH")); + builder.setNumBytes(res.getLong(COL_PARTITION_BYTES)); + builder.setNumFiles(res.getLong(COL_PARTITION_FILES)); + + partitions.add(builder.build()); + }
          + } catch (TajoException se) { + throw new TajoInternalError(se); + }

          catch (SQLException se)

          { + throw new TajoInternalError(se); + }

          finally

          { + CatalogUtil.closeQuietly(pstmt, res); + }

          +
          + return partitions;
          + }
          +
          + /**
          + * Create a select statement and parameters for querying partitions and partition keys in CatalogStore.
          + *
          + * For example, consider you have a partitioned table for three columns (i.e., col1, col2, col3).
          + * Assume that an user gives a condition WHERE (col1 ='1' or col1 = '100') and col3 > 20.
          + * There is no filter condition corresponding to col2.
          + *
          + * Then, the sql would be generated as following:
          + *
          + * SELECT A.PARTITION_ID, A.PARTITION_NAME, A.PATH FROM PARTITIONS A
          + * WHERE A.TID = ?
          + * AND A.PARTITION_ID IN (
          + * SELECT T1.PARTITION_ID FROM PARTITION_KEYS T1
          + * JOIN PARTITION_KEYS T2 ON T1.TID=T2.TID AND T1.PARTITION_ID = T2.PARTITION_ID AND T2.TID = ?
          + * AND ( T2.COLUMN_NAME = 'col2' AND T2.PARTITION_VALUE IS NOT NULL )
          + * JOIN PARTITION_KEYS T3 ON T1.TID=T3.TID AND T1.PARTITION_ID = T3.PARTITION_ID AND T3.TID = ?
          + * AND ( T3.COLUMN_NAME = 'col3' AND T3.PARTITION_VALUE > ? )
          + * WHERE T1.TID = ? AND ( T1.COLUMN_NAME = 'col1' AND T1.PARTITION_VALUE = ? )
          + * OR ( T1.COLUMN_NAME = 'col1' AND T1.PARTITION_VALUE = ? )
          — End diff –

          I also think that self-join on partition_keys is expensive way. But it might need more time to improve the select statement and according to the need, we would change partition schema. So, I want to handle this issue at another issue. What do you think about my opinion? For your information, you can see the performance of this method at https://github.com/apache/tajo/pull/624#issuecomment-127275276 .

          Show
          githubbot ASF GitHub Bot added a comment - Github user blrunner commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r40060940 — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java — @@ -2228,6 +2244,257 @@ private void setPartitionKeys(int pid, PartitionDescProto.Builder partitionDesc) return partitions; } + /** + * Check if list of partitions exist on catalog. + * + * + * @param databaseId + * @param tableId + * @return + */ + public boolean existPartitionsOnCatalog(int tableId) { + Connection conn = null; + ResultSet res = null; + PreparedStatement pstmt = null; + boolean result = false; + + try { + String sql = "SELECT COUNT CNT FROM " + + TB_PARTTIONS +" WHERE " + COL_TABLES_PK + " = ? "; + + if (LOG.isDebugEnabled()) { + LOG.debug(sql); + } + + conn = getConnection(); + pstmt = conn.prepareStatement(sql); + pstmt.setInt(1, tableId); + res = pstmt.executeQuery(); + + if (res.next()) { + if (res.getInt("CNT") > 0) { + result = true; + } + } + } catch (SQLException se) { + throw new TajoInternalError(se); + } finally { + CatalogUtil.closeQuietly(pstmt, res); + } + return result; + } + + @Override + public List<PartitionDescProto> getPartitionsByFilter(PartitionsByFilterProto request) { + throw new TajoRuntimeException(new UnsupportedException()); + } + + @Override + public List<PartitionDescProto> getPartitionsByAlgebra(PartitionsByAlgebraProto request) throws + UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException, + PartitionNotFoundException, UnsupportedException { + Connection conn = null; + PreparedStatement pstmt = null; + ResultSet res = null; + int currentIndex = 1; + String selectStatement = null; + + List<PartitionDescProto> partitions = TUtil.newList(); + List<PartitionFilterSet> filterSets = TUtil.newList(); + + int databaseId = getDatabaseId(request.getDatabaseName()); + int tableId = getTableId(databaseId, request.getDatabaseName(), request.getTableName()); + if (!existPartitionMethod(request.getDatabaseName(), request.getTableName())) { + throw new UndefinedPartitionMethodException(request.getTableName()); + } + + if (!existPartitionsOnCatalog(tableId)) { + throw new PartitionNotFoundException(request.getTableName()); + } + + try { + TableDescProto tableDesc = getTable(request.getDatabaseName(), request.getTableName()); + + selectStatement = getSelectStatementForPartitions(tableDesc.getTableName(), tableDesc.getPartition() + .getExpressionSchema().getFieldsList(), request.getAlgebra(), filterSets); + + conn = getConnection(); + pstmt = conn.prepareStatement(selectStatement); + + // Set table id by force because first parameter of all direct sql is table id + pstmt.setInt(currentIndex, tableId); + currentIndex++; + + for (PartitionFilterSet filter : filterSets) { + // Set table id by force because all filters have table id as first parameter. + pstmt.setInt(currentIndex, tableId); + currentIndex++; + + for (Pair<Type, Object> parameter : filter.getParameters()) { + switch (parameter.getFirst()) { + case BOOLEAN: + pstmt.setBoolean(currentIndex, (Boolean)parameter.getSecond()); + break; + case INT8: + pstmt.setLong(currentIndex, (Long) parameter.getSecond()); + break; + case FLOAT8: + pstmt.setDouble(currentIndex, (Double) parameter.getSecond()); + break; + case DATE: + pstmt.setDate(currentIndex, (Date) parameter.getSecond()); + break; + case TIMESTAMP: + pstmt.setTimestamp(currentIndex, (Timestamp) parameter.getSecond()); + break; + case TIME: + pstmt.setTime(currentIndex, (Time) parameter.getSecond()); + break; + default: + pstmt.setString(currentIndex, (String) parameter.getSecond()); + break; + } + currentIndex++; + } + } + + res = pstmt.executeQuery(); + + while (res.next()) { + PartitionDescProto.Builder builder = PartitionDescProto.newBuilder(); + + builder.setId(res.getInt(COL_PARTITIONS_PK)); + builder.setPartitionName(res.getString("PARTITION_NAME")); + builder.setPath(res.getString("PATH")); + builder.setNumBytes(res.getLong(COL_PARTITION_BYTES)); + builder.setNumFiles(res.getLong(COL_PARTITION_FILES)); + + partitions.add(builder.build()); + } + } catch (TajoException se) { + throw new TajoInternalError(se); + } catch (SQLException se) { + throw new TajoInternalError(se); + } finally { + CatalogUtil.closeQuietly(pstmt, res); + } + + return partitions; + } + + /** + * Create a select statement and parameters for querying partitions and partition keys in CatalogStore. + * + * For example, consider you have a partitioned table for three columns (i.e., col1, col2, col3). + * Assume that an user gives a condition WHERE (col1 ='1' or col1 = '100') and col3 > 20. + * There is no filter condition corresponding to col2. + * + * Then, the sql would be generated as following: + * + * SELECT A.PARTITION_ID, A.PARTITION_NAME, A.PATH FROM PARTITIONS A + * WHERE A.TID = ? + * AND A.PARTITION_ID IN ( + * SELECT T1.PARTITION_ID FROM PARTITION_KEYS T1 + * JOIN PARTITION_KEYS T2 ON T1.TID=T2.TID AND T1.PARTITION_ID = T2.PARTITION_ID AND T2.TID = ? + * AND ( T2.COLUMN_NAME = 'col2' AND T2.PARTITION_VALUE IS NOT NULL ) + * JOIN PARTITION_KEYS T3 ON T1.TID=T3.TID AND T1.PARTITION_ID = T3.PARTITION_ID AND T3.TID = ? + * AND ( T3.COLUMN_NAME = 'col3' AND T3.PARTITION_VALUE > ? ) + * WHERE T1.TID = ? AND ( T1.COLUMN_NAME = 'col1' AND T1.PARTITION_VALUE = ? ) + * OR ( T1.COLUMN_NAME = 'col1' AND T1.PARTITION_VALUE = ? ) — End diff – I also think that self-join on partition_keys is expensive way. But it might need more time to improve the select statement and according to the need, we would change partition schema. So, I want to handle this issue at another issue. What do you think about my opinion? For your information, you can see the performance of this method at https://github.com/apache/tajo/pull/624#issuecomment-127275276 .
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user blrunner commented on the pull request:

          https://github.com/apache/tajo/pull/772#issuecomment-142211185

          @jihoonson

          Thank you for your detailed review. I reflected your comments and left some comments.

          Show
          githubbot ASF GitHub Bot added a comment - Github user blrunner commented on the pull request: https://github.com/apache/tajo/pull/772#issuecomment-142211185 @jihoonson Thank you for your detailed review. I reflected your comments and left some comments.
          Hide
          tajoqa Tajo QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12761600/TAJO-1493_5.patch
          against master revision release-0.9.0-rc0-496-gcf3eef9.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

          +1 checkstyle. The patch generated 0 code style errors.

          -1 findbugs. The patch appears to cause Findbugs (version 2.0.3) to fail.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in tajo-catalog/tajo-catalog-client tajo-catalog/tajo-catalog-common tajo-catalog/tajo-catalog-drivers/tajo-hive tajo-catalog/tajo-catalog-server tajo-cli tajo-client tajo-common tajo-core tajo-core-tests tajo-plan tajo-storage/tajo-storage-hdfs:
          org.apache.tajo.engine.planner.TestEvalNodeToExprConverter

          Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/870//testReport/
          Findbugs results: https://builds.apache.org/job/PreCommit-TAJO-Build/870//findbugsResult
          Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/870//console

          This message is automatically generated.

          Show
          tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12761600/TAJO-1493_5.patch against master revision release-0.9.0-rc0-496-gcf3eef9. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. -1 findbugs. The patch appears to cause Findbugs (version 2.0.3) to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in tajo-catalog/tajo-catalog-client tajo-catalog/tajo-catalog-common tajo-catalog/tajo-catalog-drivers/tajo-hive tajo-catalog/tajo-catalog-server tajo-cli tajo-client tajo-common tajo-core tajo-core-tests tajo-plan tajo-storage/tajo-storage-hdfs: org.apache.tajo.engine.planner.TestEvalNodeToExprConverter Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/870//testReport/ Findbugs results: https://builds.apache.org/job/PreCommit-TAJO-Build/870//findbugsResult Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/870//console This message is automatically generated.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r40093053

          — Diff: tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java —
          @@ -812,16 +812,34 @@ public static AlterTableDesc setProperty(String tableName, KeyValueSet params, A
          /**

          • Converts passed parameters to a AlterTableDesc. This method would be called when adding a partition or dropping
          • a table. This creates AlterTableDesc that is a wrapper class for protocol buffer.
            + * *
            + * @param tableName
            + * @param columns
            + * @param values
            + * @param path
            + * @param alterTableType
            + * @return
            + */
            + public static AlterTableDesc addOrDropPartition(String tableName, String[] columns,
            + String[] values, @Nullable String path, AlterTableType
            + alterTableType) { + + return addOrDropPartition(tableName, columns, values, path, alterTableType, 0L); + }

            + /**
            + * Converts passed parameters to a AlterTableDesc. This method would be called when adding a partition or dropping
            + * a table. This creates AlterTableDesc that is a wrapper class for protocol buffer.
            *

          • @param tableName table name
          • @param columns partition column names
          • @param values partition values
          • * @param location partition location
            + * @param path partition directory path
          • @param alterTableType ADD_PARTITION or DROP_PARTITION
            + * @param numBytes contents length
          • @return AlterTableDesc
            */
            public static AlterTableDesc addOrDropPartition(String tableName, String[] columns,
          • String[] values, String location, AlterTableType alterTableType) {
            + String[] values, @Nullable String path, AlterTableType alterTableType, long numBytes) {
              • End diff –

          Please fix indentation.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r40093053 — Diff: tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java — @@ -812,16 +812,34 @@ public static AlterTableDesc setProperty(String tableName, KeyValueSet params, A /** Converts passed parameters to a AlterTableDesc. This method would be called when adding a partition or dropping a table. This creates AlterTableDesc that is a wrapper class for protocol buffer. + * * + * @param tableName + * @param columns + * @param values + * @param path + * @param alterTableType + * @return + */ + public static AlterTableDesc addOrDropPartition(String tableName, String[] columns, + String[] values, @Nullable String path, AlterTableType + alterTableType) { + + return addOrDropPartition(tableName, columns, values, path, alterTableType, 0L); + } + /** + * Converts passed parameters to a AlterTableDesc. This method would be called when adding a partition or dropping + * a table. This creates AlterTableDesc that is a wrapper class for protocol buffer. * @param tableName table name @param columns partition column names @param values partition values * @param location partition location + * @param path partition directory path @param alterTableType ADD_PARTITION or DROP_PARTITION + * @param numBytes contents length @return AlterTableDesc */ public static AlterTableDesc addOrDropPartition(String tableName, String[] columns, String[] values, String location, AlterTableType alterTableType) { + String[] values, @Nullable String path, AlterTableType alterTableType, long numBytes) { End diff – Please fix indentation.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r40093055

          — Diff: tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java —
          @@ -812,16 +812,34 @@ public static AlterTableDesc setProperty(String tableName, KeyValueSet params, A
          /**

          • Converts passed parameters to a AlterTableDesc. This method would be called when adding a partition or dropping
          • a table. This creates AlterTableDesc that is a wrapper class for protocol buffer.
            + * *
            + * @param tableName
            + * @param columns
            + * @param values
            + * @param path
            + * @param alterTableType
            + * @return
            + */
            + public static AlterTableDesc addOrDropPartition(String tableName, String[] columns,
            + String[] values, @Nullable String path, AlterTableType
            + alterTableType) {
              • End diff –

          Please fix indentation.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r40093055 — Diff: tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java — @@ -812,16 +812,34 @@ public static AlterTableDesc setProperty(String tableName, KeyValueSet params, A /** Converts passed parameters to a AlterTableDesc. This method would be called when adding a partition or dropping a table. This creates AlterTableDesc that is a wrapper class for protocol buffer. + * * + * @param tableName + * @param columns + * @param values + * @param path + * @param alterTableType + * @return + */ + public static AlterTableDesc addOrDropPartition(String tableName, String[] columns, + String[] values, @Nullable String path, AlterTableType + alterTableType) { End diff – Please fix indentation.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r40093128

          — Diff: tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto —
          @@ -262,6 +262,18 @@ message PartitionIdentifierProto

          { optional string partitionName = 3; }

          +message PartitionsByAlgebraProto {
          + required string databaseName = 1;
          + required string tableName = 2;
          + required string algebra = 3; /** json object which contains algebra expressions */
          — End diff –

          Please use ```//``` instead of ```/**/```.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r40093128 — Diff: tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto — @@ -262,6 +262,18 @@ message PartitionIdentifierProto { optional string partitionName = 3; } +message PartitionsByAlgebraProto { + required string databaseName = 1; + required string tableName = 2; + required string algebra = 3; /** json object which contains algebra expressions */ — End diff – Please use ```//``` instead of ```/**/```.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r40093205

          — Diff: tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto —
          @@ -262,6 +262,18 @@ message PartitionIdentifierProto

          { optional string partitionName = 3; }

          +message PartitionsByAlgebraProto

          { + required string databaseName = 1; + required string tableName = 2; + required string algebra = 3; /** json object which contains algebra expressions */ +}

          +
          +message PartitionsByFilterProto {
          + required string databaseName = 1;
          + required string tableName = 2;
          + required string filter = 3; /** filter condition, for example col1 = 'TAJO' and col2 = 100. */
          — End diff –

          Please use ```//``` instead of ```/**/```.
          In addition, which format is used for filter serialization?

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r40093205 — Diff: tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto — @@ -262,6 +262,18 @@ message PartitionIdentifierProto { optional string partitionName = 3; } +message PartitionsByAlgebraProto { + required string databaseName = 1; + required string tableName = 2; + required string algebra = 3; /** json object which contains algebra expressions */ +} + +message PartitionsByFilterProto { + required string databaseName = 1; + required string tableName = 2; + required string filter = 3; /** filter condition, for example col1 = 'TAJO' and col2 = 100. */ — End diff – Please use ```//``` instead of ```/**/```. In addition, which format is used for filter serialization?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r40093853

          — Diff: tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogStore.java —
          @@ -845,13 +856,174 @@ public boolean existPartitionMethod(String databaseName, String tableName) throw
          }

          @Override

          • public List<CatalogProtos.PartitionDescProto> getPartitions(String databaseName,
          • String tableName) {
          • throw new UnsupportedOperationException();
            + public List<CatalogProtos.PartitionDescProto> getPartitionsOfTable(String databaseName, String tableName)
            + throws UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException { + PartitionsByFilterProto.Builder request = PartitionsByFilterProto.newBuilder(); + request.setDatabaseName(databaseName); + request.setTableName(tableName); + request.setFilter(""); + + return getPartitionsByFilter(request.build()); + }

            +
            + @Override
            + public List<PartitionDescProto> getPartitionsByAlgebra(PartitionsByAlgebraProto request) throws
            + UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException, UnsupportedException {
            +
            + List<PartitionDescProto> list = null;
            +
            + try

            Unknown macro: { + String databaseName = request.getDatabaseName(); + String tableName = request.getTableName(); + + if (!existDatabase(databaseName)) { + throw new UndefinedDatabaseException(tableName); + } + + if (!existTable(databaseName, tableName)) { + throw new UndefinedTableException(tableName); + } + + if (!existPartitionMethod(databaseName, tableName)) { + throw new UndefinedPartitionMethodException(tableName); + } + + TableDescProto tableDesc = getTable(databaseName, tableName); + String filter = getFilter(databaseName, tableName, tableDesc.getPartition().getExpressionSchema().getFieldsList() + , request.getAlgebra()); + list = getPartitionsByFilterFromHiveMetaStore(databaseName, tableName, filter); + }

            catch (UnsupportedException ue)

            { + throw ue; + }

            catch (Exception se)

            { + throw new TajoInternalError(se); + }

            +
            + return list;
            + }
            +
            + private String getFilter(String databaseName, String tableName, List<ColumnProto> partitionColumns
            + , String json) throws TajoException {
            +
            + Expr[] exprs = null;
            +
            + if (json != null && !json.isEmpty())

            { + Expr algebra = JsonHelper.fromJson(json, Expr.class); + exprs = AlgebraicUtil.toConjunctiveNormalFormArray(algebra); + }

            +
            + PartitionFilterAlgebraVisitor visitor = new PartitionFilterAlgebraVisitor();
            + visitor.setIsHiveCatalog(true);
            +
            + Expr[] filters = AlgebraicUtil.getRearrangedCNFExpressions(databaseName + "." + tableName, partitionColumns, exprs);
            +
            + StringBuffer sb = new StringBuffer();
            +
            + // Write join clause from second column to last column.
            + Column target;
            +
            + int addedFilter = 0;
            + String result;
            + for (int i = 0; i < partitionColumns.size(); i++) {
            + target = new Column(partitionColumns.get);
            +
            + if (!(filters[i] instanceof IsNullPredicate)) {
            + visitor.setColumn(target);
            + visitor.visit(null, new Stack<Expr>(), filters[i]);
            + result = visitor.getResult();
            +
            + // If visitor build filter successfully, add filter to be used for executing hive api.
            + if (result.length() > 0) {
            + if (addedFilter > 0) {

              • End diff –

          ```addedFilter``` can be replaced with sb.length().

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r40093853 — Diff: tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogStore.java — @@ -845,13 +856,174 @@ public boolean existPartitionMethod(String databaseName, String tableName) throw } @Override public List<CatalogProtos.PartitionDescProto> getPartitions(String databaseName, String tableName) { throw new UnsupportedOperationException(); + public List<CatalogProtos.PartitionDescProto> getPartitionsOfTable(String databaseName, String tableName) + throws UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException { + PartitionsByFilterProto.Builder request = PartitionsByFilterProto.newBuilder(); + request.setDatabaseName(databaseName); + request.setTableName(tableName); + request.setFilter(""); + + return getPartitionsByFilter(request.build()); + } + + @Override + public List<PartitionDescProto> getPartitionsByAlgebra(PartitionsByAlgebraProto request) throws + UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException, UnsupportedException { + + List<PartitionDescProto> list = null; + + try Unknown macro: { + String databaseName = request.getDatabaseName(); + String tableName = request.getTableName(); + + if (!existDatabase(databaseName)) { + throw new UndefinedDatabaseException(tableName); + } + + if (!existTable(databaseName, tableName)) { + throw new UndefinedTableException(tableName); + } + + if (!existPartitionMethod(databaseName, tableName)) { + throw new UndefinedPartitionMethodException(tableName); + } + + TableDescProto tableDesc = getTable(databaseName, tableName); + String filter = getFilter(databaseName, tableName, tableDesc.getPartition().getExpressionSchema().getFieldsList() + , request.getAlgebra()); + list = getPartitionsByFilterFromHiveMetaStore(databaseName, tableName, filter); + } catch (UnsupportedException ue) { + throw ue; + } catch (Exception se) { + throw new TajoInternalError(se); + } + + return list; + } + + private String getFilter(String databaseName, String tableName, List<ColumnProto> partitionColumns + , String json) throws TajoException { + + Expr[] exprs = null; + + if (json != null && !json.isEmpty()) { + Expr algebra = JsonHelper.fromJson(json, Expr.class); + exprs = AlgebraicUtil.toConjunctiveNormalFormArray(algebra); + } + + PartitionFilterAlgebraVisitor visitor = new PartitionFilterAlgebraVisitor(); + visitor.setIsHiveCatalog(true); + + Expr[] filters = AlgebraicUtil.getRearrangedCNFExpressions(databaseName + "." + tableName, partitionColumns, exprs); + + StringBuffer sb = new StringBuffer(); + + // Write join clause from second column to last column. + Column target; + + int addedFilter = 0; + String result; + for (int i = 0; i < partitionColumns.size(); i++) { + target = new Column(partitionColumns.get ); + + if (!(filters [i] instanceof IsNullPredicate)) { + visitor.setColumn(target); + visitor.visit(null, new Stack<Expr>(), filters [i] ); + result = visitor.getResult(); + + // If visitor build filter successfully, add filter to be used for executing hive api. + if (result.length() > 0) { + if (addedFilter > 0) { End diff – ```addedFilter``` can be replaced with sb.length().
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r40095339

          — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java —
          @@ -2228,6 +2238,260 @@ private void setPartitionKeys(int pid, PartitionDescProto.Builder partitionDesc)
          return partitions;
          }

          + /**
          + * Check if list of partitions exist on catalog.
          + *
          + *
          + * @param databaseId
          — End diff –

          Wrong parameter.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r40095339 — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java — @@ -2228,6 +2238,260 @@ private void setPartitionKeys(int pid, PartitionDescProto.Builder partitionDesc) return partitions; } + /** + * Check if list of partitions exist on catalog. + * + * + * @param databaseId — End diff – Wrong parameter.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r40095995

          — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java —
          @@ -2228,6 +2244,257 @@ private void setPartitionKeys(int pid, PartitionDescProto.Builder partitionDesc)
          return partitions;
          }

          + /**
          + * Check if list of partitions exist on catalog.
          + *
          + *
          + * @param databaseId
          + * @param tableId
          + * @return
          + */
          + public boolean existPartitionsOnCatalog(int tableId) {
          + Connection conn = null;
          + ResultSet res = null;
          + PreparedStatement pstmt = null;
          + boolean result = false;
          +
          + try {
          + String sql = "SELECT COUNT CNT FROM "
          + + TB_PARTTIONS +" WHERE " + COL_TABLES_PK + " = ? ";
          +
          + if (LOG.isDebugEnabled())

          { + LOG.debug(sql); + }

          +
          + conn = getConnection();
          + pstmt = conn.prepareStatement(sql);
          + pstmt.setInt(1, tableId);
          + res = pstmt.executeQuery();
          +
          + if (res.next()) {
          + if (res.getInt("CNT") > 0)

          { + result = true; + }

          + }
          + } catch (SQLException se)

          { + throw new TajoInternalError(se); + }

          finally

          { + CatalogUtil.closeQuietly(pstmt, res); + }

          + return result;
          + }
          +
          + @Override
          + public List<PartitionDescProto> getPartitionsByFilter(PartitionsByFilterProto request) {
          + throw new TajoRuntimeException(new UnsupportedException());
          — End diff –

          I got your point.
          If so, please update the comments on both ```CatalogStore::getPartitionsByAlgebra()``` and ```CatalogStore::getPartitionsByFilter()``` to let others know each method is implemented in AbstractDBStore and HiveCatalogStore, respectively.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r40095995 — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java — @@ -2228,6 +2244,257 @@ private void setPartitionKeys(int pid, PartitionDescProto.Builder partitionDesc) return partitions; } + /** + * Check if list of partitions exist on catalog. + * + * + * @param databaseId + * @param tableId + * @return + */ + public boolean existPartitionsOnCatalog(int tableId) { + Connection conn = null; + ResultSet res = null; + PreparedStatement pstmt = null; + boolean result = false; + + try { + String sql = "SELECT COUNT CNT FROM " + + TB_PARTTIONS +" WHERE " + COL_TABLES_PK + " = ? "; + + if (LOG.isDebugEnabled()) { + LOG.debug(sql); + } + + conn = getConnection(); + pstmt = conn.prepareStatement(sql); + pstmt.setInt(1, tableId); + res = pstmt.executeQuery(); + + if (res.next()) { + if (res.getInt("CNT") > 0) { + result = true; + } + } + } catch (SQLException se) { + throw new TajoInternalError(se); + } finally { + CatalogUtil.closeQuietly(pstmt, res); + } + return result; + } + + @Override + public List<PartitionDescProto> getPartitionsByFilter(PartitionsByFilterProto request) { + throw new TajoRuntimeException(new UnsupportedException()); — End diff – I got your point. If so, please update the comments on both ```CatalogStore::getPartitionsByAlgebra()``` and ```CatalogStore::getPartitionsByFilter()``` to let others know each method is implemented in AbstractDBStore and HiveCatalogStore, respectively.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r40096556

          — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java —
          @@ -2228,6 +2244,257 @@ private void setPartitionKeys(int pid, PartitionDescProto.Builder partitionDesc)
          return partitions;
          }

          + /**
          + * Check if list of partitions exist on catalog.
          + *
          + *
          + * @param databaseId
          + * @param tableId
          + * @return
          + */
          + public boolean existPartitionsOnCatalog(int tableId) {
          + Connection conn = null;
          + ResultSet res = null;
          + PreparedStatement pstmt = null;
          + boolean result = false;
          +
          + try {
          + String sql = "SELECT COUNT CNT FROM "
          + + TB_PARTTIONS +" WHERE " + COL_TABLES_PK + " = ? ";
          +
          + if (LOG.isDebugEnabled())

          { + LOG.debug(sql); + }

          +
          + conn = getConnection();
          + pstmt = conn.prepareStatement(sql);
          + pstmt.setInt(1, tableId);
          + res = pstmt.executeQuery();
          +
          + if (res.next()) {
          + if (res.getInt("CNT") > 0)

          { + result = true; + }

          + }
          + } catch (SQLException se)

          { + throw new TajoInternalError(se); + }

          finally

          { + CatalogUtil.closeQuietly(pstmt, res); + }

          + return result;
          + }
          +
          + @Override
          + public List<PartitionDescProto> getPartitionsByFilter(PartitionsByFilterProto request) {
          + throw new TajoRuntimeException(new UnsupportedException());
          — End diff –

          And please open a jira ticket for this issue.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r40096556 — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java — @@ -2228,6 +2244,257 @@ private void setPartitionKeys(int pid, PartitionDescProto.Builder partitionDesc) return partitions; } + /** + * Check if list of partitions exist on catalog. + * + * + * @param databaseId + * @param tableId + * @return + */ + public boolean existPartitionsOnCatalog(int tableId) { + Connection conn = null; + ResultSet res = null; + PreparedStatement pstmt = null; + boolean result = false; + + try { + String sql = "SELECT COUNT CNT FROM " + + TB_PARTTIONS +" WHERE " + COL_TABLES_PK + " = ? "; + + if (LOG.isDebugEnabled()) { + LOG.debug(sql); + } + + conn = getConnection(); + pstmt = conn.prepareStatement(sql); + pstmt.setInt(1, tableId); + res = pstmt.executeQuery(); + + if (res.next()) { + if (res.getInt("CNT") > 0) { + result = true; + } + } + } catch (SQLException se) { + throw new TajoInternalError(se); + } finally { + CatalogUtil.closeQuietly(pstmt, res); + } + return result; + } + + @Override + public List<PartitionDescProto> getPartitionsByFilter(PartitionsByFilterProto request) { + throw new TajoRuntimeException(new UnsupportedException()); — End diff – And please open a jira ticket for this issue.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r40096584

          — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java —
          @@ -2228,6 +2244,257 @@ private void setPartitionKeys(int pid, PartitionDescProto.Builder partitionDesc)
          return partitions;
          }

          + /**
          + * Check if list of partitions exist on catalog.
          + *
          + *
          + * @param databaseId
          + * @param tableId
          + * @return
          + */
          + public boolean existPartitionsOnCatalog(int tableId) {
          + Connection conn = null;
          + ResultSet res = null;
          + PreparedStatement pstmt = null;
          + boolean result = false;
          +
          + try {
          + String sql = "SELECT COUNT CNT FROM "
          + + TB_PARTTIONS +" WHERE " + COL_TABLES_PK + " = ? ";
          +
          + if (LOG.isDebugEnabled())

          { + LOG.debug(sql); + }

          +
          + conn = getConnection();
          + pstmt = conn.prepareStatement(sql);
          + pstmt.setInt(1, tableId);
          + res = pstmt.executeQuery();
          +
          + if (res.next()) {
          + if (res.getInt("CNT") > 0)

          { + result = true; + }

          + }
          + } catch (SQLException se)

          { + throw new TajoInternalError(se); + } finally { + CatalogUtil.closeQuietly(pstmt, res); + }
          + return result;
          + }
          +
          + @Override
          + public List<PartitionDescProto> getPartitionsByFilter(PartitionsByFilterProto request) { + throw new TajoRuntimeException(new UnsupportedException()); + }
          +
          + @Override
          + public List<PartitionDescProto> getPartitionsByAlgebra(PartitionsByAlgebraProto request) throws
          + UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException,
          + PartitionNotFoundException, UnsupportedException {
          + Connection conn = null;
          + PreparedStatement pstmt = null;
          + ResultSet res = null;
          + int currentIndex = 1;
          + String selectStatement = null;
          +
          + List<PartitionDescProto> partitions = TUtil.newList();
          + List<PartitionFilterSet> filterSets = TUtil.newList();
          +
          + int databaseId = getDatabaseId(request.getDatabaseName());
          + int tableId = getTableId(databaseId, request.getDatabaseName(), request.getTableName());
          + if (!existPartitionMethod(request.getDatabaseName(), request.getTableName())) { + throw new UndefinedPartitionMethodException(request.getTableName()); + }
          +
          + if (!existPartitionsOnCatalog(tableId)) { + throw new PartitionNotFoundException(request.getTableName()); + }
          +
          + try {
          + TableDescProto tableDesc = getTable(request.getDatabaseName(), request.getTableName());
          +
          + selectStatement = getSelectStatementForPartitions(tableDesc.getTableName(), tableDesc.getPartition()
          + .getExpressionSchema().getFieldsList(), request.getAlgebra(), filterSets);
          +
          + conn = getConnection();
          + pstmt = conn.prepareStatement(selectStatement);
          +
          + // Set table id by force because first parameter of all direct sql is table id
          + pstmt.setInt(currentIndex, tableId);
          + currentIndex++;
          +
          + for (PartitionFilterSet filter : filterSets) {
          + // Set table id by force because all filters have table id as first parameter.
          + pstmt.setInt(currentIndex, tableId);
          + currentIndex++;
          +
          + for (Pair<Type, Object> parameter : filter.getParameters()) {
          + switch (parameter.getFirst()) { + case BOOLEAN: + pstmt.setBoolean(currentIndex, (Boolean)parameter.getSecond()); + break; + case INT8: + pstmt.setLong(currentIndex, (Long) parameter.getSecond()); + break; + case FLOAT8: + pstmt.setDouble(currentIndex, (Double) parameter.getSecond()); + break; + case DATE: + pstmt.setDate(currentIndex, (Date) parameter.getSecond()); + break; + case TIMESTAMP: + pstmt.setTimestamp(currentIndex, (Timestamp) parameter.getSecond()); + break; + case TIME: + pstmt.setTime(currentIndex, (Time) parameter.getSecond()); + break; + default: + pstmt.setString(currentIndex, (String) parameter.getSecond()); + break; + }
          + currentIndex++;
          + }
          + }
          +
          + res = pstmt.executeQuery();
          +
          + while (res.next()) { + PartitionDescProto.Builder builder = PartitionDescProto.newBuilder(); + + builder.setId(res.getInt(COL_PARTITIONS_PK)); + builder.setPartitionName(res.getString("PARTITION_NAME")); + builder.setPath(res.getString("PATH")); + builder.setNumBytes(res.getLong(COL_PARTITION_BYTES)); + builder.setNumFiles(res.getLong(COL_PARTITION_FILES)); + + partitions.add(builder.build()); + }
          + } catch (TajoException se) { + throw new TajoInternalError(se); + }

          catch (SQLException se)

          { + throw new TajoInternalError(se); + }

          finally

          { + CatalogUtil.closeQuietly(pstmt, res); + }

          +
          + return partitions;
          + }
          +
          + /**
          + * Create a select statement and parameters for querying partitions and partition keys in CatalogStore.
          + *
          + * For example, consider you have a partitioned table for three columns (i.e., col1, col2, col3).
          + * Assume that an user gives a condition WHERE (col1 ='1' or col1 = '100') and col3 > 20.
          + * There is no filter condition corresponding to col2.
          + *
          + * Then, the sql would be generated as following:
          + *
          + * SELECT A.PARTITION_ID, A.PARTITION_NAME, A.PATH FROM PARTITIONS A
          + * WHERE A.TID = ?
          + * AND A.PARTITION_ID IN (
          + * SELECT T1.PARTITION_ID FROM PARTITION_KEYS T1
          + * JOIN PARTITION_KEYS T2 ON T1.TID=T2.TID AND T1.PARTITION_ID = T2.PARTITION_ID AND T2.TID = ?
          + * AND ( T2.COLUMN_NAME = 'col2' AND T2.PARTITION_VALUE IS NOT NULL )
          + * JOIN PARTITION_KEYS T3 ON T1.TID=T3.TID AND T1.PARTITION_ID = T3.PARTITION_ID AND T3.TID = ?
          + * AND ( T3.COLUMN_NAME = 'col3' AND T3.PARTITION_VALUE > ? )
          + * WHERE T1.TID = ? AND ( T1.COLUMN_NAME = 'col1' AND T1.PARTITION_VALUE = ? )
          + * OR ( T1.COLUMN_NAME = 'col1' AND T1.PARTITION_VALUE = ? )
          — End diff –

          Ok. Please open a jira ticket for this issue.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r40096584 — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java — @@ -2228,6 +2244,257 @@ private void setPartitionKeys(int pid, PartitionDescProto.Builder partitionDesc) return partitions; } + /** + * Check if list of partitions exist on catalog. + * + * + * @param databaseId + * @param tableId + * @return + */ + public boolean existPartitionsOnCatalog(int tableId) { + Connection conn = null; + ResultSet res = null; + PreparedStatement pstmt = null; + boolean result = false; + + try { + String sql = "SELECT COUNT CNT FROM " + + TB_PARTTIONS +" WHERE " + COL_TABLES_PK + " = ? "; + + if (LOG.isDebugEnabled()) { + LOG.debug(sql); + } + + conn = getConnection(); + pstmt = conn.prepareStatement(sql); + pstmt.setInt(1, tableId); + res = pstmt.executeQuery(); + + if (res.next()) { + if (res.getInt("CNT") > 0) { + result = true; + } + } + } catch (SQLException se) { + throw new TajoInternalError(se); + } finally { + CatalogUtil.closeQuietly(pstmt, res); + } + return result; + } + + @Override + public List<PartitionDescProto> getPartitionsByFilter(PartitionsByFilterProto request) { + throw new TajoRuntimeException(new UnsupportedException()); + } + + @Override + public List<PartitionDescProto> getPartitionsByAlgebra(PartitionsByAlgebraProto request) throws + UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException, + PartitionNotFoundException, UnsupportedException { + Connection conn = null; + PreparedStatement pstmt = null; + ResultSet res = null; + int currentIndex = 1; + String selectStatement = null; + + List<PartitionDescProto> partitions = TUtil.newList(); + List<PartitionFilterSet> filterSets = TUtil.newList(); + + int databaseId = getDatabaseId(request.getDatabaseName()); + int tableId = getTableId(databaseId, request.getDatabaseName(), request.getTableName()); + if (!existPartitionMethod(request.getDatabaseName(), request.getTableName())) { + throw new UndefinedPartitionMethodException(request.getTableName()); + } + + if (!existPartitionsOnCatalog(tableId)) { + throw new PartitionNotFoundException(request.getTableName()); + } + + try { + TableDescProto tableDesc = getTable(request.getDatabaseName(), request.getTableName()); + + selectStatement = getSelectStatementForPartitions(tableDesc.getTableName(), tableDesc.getPartition() + .getExpressionSchema().getFieldsList(), request.getAlgebra(), filterSets); + + conn = getConnection(); + pstmt = conn.prepareStatement(selectStatement); + + // Set table id by force because first parameter of all direct sql is table id + pstmt.setInt(currentIndex, tableId); + currentIndex++; + + for (PartitionFilterSet filter : filterSets) { + // Set table id by force because all filters have table id as first parameter. + pstmt.setInt(currentIndex, tableId); + currentIndex++; + + for (Pair<Type, Object> parameter : filter.getParameters()) { + switch (parameter.getFirst()) { + case BOOLEAN: + pstmt.setBoolean(currentIndex, (Boolean)parameter.getSecond()); + break; + case INT8: + pstmt.setLong(currentIndex, (Long) parameter.getSecond()); + break; + case FLOAT8: + pstmt.setDouble(currentIndex, (Double) parameter.getSecond()); + break; + case DATE: + pstmt.setDate(currentIndex, (Date) parameter.getSecond()); + break; + case TIMESTAMP: + pstmt.setTimestamp(currentIndex, (Timestamp) parameter.getSecond()); + break; + case TIME: + pstmt.setTime(currentIndex, (Time) parameter.getSecond()); + break; + default: + pstmt.setString(currentIndex, (String) parameter.getSecond()); + break; + } + currentIndex++; + } + } + + res = pstmt.executeQuery(); + + while (res.next()) { + PartitionDescProto.Builder builder = PartitionDescProto.newBuilder(); + + builder.setId(res.getInt(COL_PARTITIONS_PK)); + builder.setPartitionName(res.getString("PARTITION_NAME")); + builder.setPath(res.getString("PATH")); + builder.setNumBytes(res.getLong(COL_PARTITION_BYTES)); + builder.setNumFiles(res.getLong(COL_PARTITION_FILES)); + + partitions.add(builder.build()); + } + } catch (TajoException se) { + throw new TajoInternalError(se); + } catch (SQLException se) { + throw new TajoInternalError(se); + } finally { + CatalogUtil.closeQuietly(pstmt, res); + } + + return partitions; + } + + /** + * Create a select statement and parameters for querying partitions and partition keys in CatalogStore. + * + * For example, consider you have a partitioned table for three columns (i.e., col1, col2, col3). + * Assume that an user gives a condition WHERE (col1 ='1' or col1 = '100') and col3 > 20. + * There is no filter condition corresponding to col2. + * + * Then, the sql would be generated as following: + * + * SELECT A.PARTITION_ID, A.PARTITION_NAME, A.PATH FROM PARTITIONS A + * WHERE A.TID = ? + * AND A.PARTITION_ID IN ( + * SELECT T1.PARTITION_ID FROM PARTITION_KEYS T1 + * JOIN PARTITION_KEYS T2 ON T1.TID=T2.TID AND T1.PARTITION_ID = T2.PARTITION_ID AND T2.TID = ? + * AND ( T2.COLUMN_NAME = 'col2' AND T2.PARTITION_VALUE IS NOT NULL ) + * JOIN PARTITION_KEYS T3 ON T1.TID=T3.TID AND T1.PARTITION_ID = T3.PARTITION_ID AND T3.TID = ? + * AND ( T3.COLUMN_NAME = 'col3' AND T3.PARTITION_VALUE > ? ) + * WHERE T1.TID = ? AND ( T1.COLUMN_NAME = 'col1' AND T1.PARTITION_VALUE = ? ) + * OR ( T1.COLUMN_NAME = 'col1' AND T1.PARTITION_VALUE = ? ) — End diff – Ok. Please open a jira ticket for this issue.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r40096748

          — Diff: tajo-catalog/tajo-catalog-server/src/main/resources/schemas/mariadb/mariadb.xml —
          @@ -150,7 +151,8 @@
          TID INT NOT NULL,
          PARTITION_NAME VARCHAR(255) BINARY,
          PATH VARCHAR(4096) BINARY,

          • UNIQUE INDEX PARTITION_UNIQUE_IDX (TID, PARTITION_NAME),
            + NUM_BYTES BIGINT,
              • End diff –

          Please fix the indentation.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r40096748 — Diff: tajo-catalog/tajo-catalog-server/src/main/resources/schemas/mariadb/mariadb.xml — @@ -150,7 +151,8 @@ TID INT NOT NULL, PARTITION_NAME VARCHAR(255) BINARY, PATH VARCHAR(4096) BINARY, UNIQUE INDEX PARTITION_UNIQUE_IDX (TID, PARTITION_NAME), + NUM_BYTES BIGINT, End diff – Please fix the indentation.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r40096792

          — Diff: tajo-catalog/tajo-catalog-server/src/main/resources/schemas/mysql/mysql.xml —
          @@ -154,7 +155,8 @@
          TID INT NOT NULL,
          PARTITION_NAME VARCHAR(255) BINARY,
          PATH VARCHAR(4096) BINARY,

          • UNIQUE INDEX PARTITION_UNIQUE_IDX (TID, PARTITION_NAME),
            + NUM_BYTES BIGINT,
              • End diff –

          Please fix the indentation.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r40096792 — Diff: tajo-catalog/tajo-catalog-server/src/main/resources/schemas/mysql/mysql.xml — @@ -154,7 +155,8 @@ TID INT NOT NULL, PARTITION_NAME VARCHAR(255) BINARY, PATH VARCHAR(4096) BINARY, UNIQUE INDEX PARTITION_UNIQUE_IDX (TID, PARTITION_NAME), + NUM_BYTES BIGINT, End diff – Please fix the indentation.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r40097101

          — Diff: tajo-plan/src/main/java/org/apache/tajo/plan/expr/AlgebraicUtil.java —
          @@ -493,4 +497,190 @@ public Object visitTimeLiteral(Object ctx, Stack<Expr> stack, TimeLiteral expr)
          return super.visitTimeLiteral(ctx, stack, expr);
          }
          }
          +
          + /**
          + * Find the top expr matched to type from the given expr
          + *
          + * @param expr start expr
          + * @param type to find
          + * @return a found expr
          + */
          + public static <T extends Expr> T findTopExpr(Expr expr, OpType type) throws TajoException {
          + Preconditions.checkNotNull(expr);
          + Preconditions.checkNotNull(type);
          +
          + ExprFinder finder = new ExprFinder(type);
          + finder.visit(null, new Stack<Expr>(), expr);
          +
          + if (finder.getFoundExprs().size() == 0)

          { + return null; + }

          + return (T) finder.getFoundExprs().get(0);
          + }
          +
          + private static class ExprFinder extends SimpleAlgebraVisitor<Object, Expr> {
          + private List<Expr> list = new ArrayList<Expr>();
          + private final OpType[] tofind;
          + private boolean topmost = false;
          + private boolean finished = false;
          +
          + public ExprFinder(OpType... type)

          { + + this.tofind = type; + }

          +
          + public ExprFinder(OpType[] type, boolean topmost)

          { + this(type); + this.topmost = topmost; + }

          +
          + @Override
          + public Expr visit(Object ctx, Stack<Expr> stack, Expr expr) throws TajoException {
          + if (!finished) {
          + for (OpType type : tofind) {
          + if (expr.getType() == type)

          { + list.add(expr); + }

          + if (topmost && list.size() > 0)

          { + finished = true; + }

          + }
          + }
          + return super.visit(ctx, stack, expr);
          + }
          +
          + public List<Expr> getFoundExprs()

          { + return list; + }

          +
          + }
          +
          + public static Expr[] toConjunctiveNormalFormArray(Expr expr)

          { + List<Expr> list = new ArrayList<Expr>(); + toConjunctiveNormalFormArrayRecursive(expr, list); + return list.toArray(new Expr[list.size()]); + }

          +
          + private static void toConjunctiveNormalFormArrayRecursive(Expr node, List<Expr> found) {
          — End diff –

          Sorry, my mistake.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r40097101 — Diff: tajo-plan/src/main/java/org/apache/tajo/plan/expr/AlgebraicUtil.java — @@ -493,4 +497,190 @@ public Object visitTimeLiteral(Object ctx, Stack<Expr> stack, TimeLiteral expr) return super.visitTimeLiteral(ctx, stack, expr); } } + + /** + * Find the top expr matched to type from the given expr + * + * @param expr start expr + * @param type to find + * @return a found expr + */ + public static <T extends Expr> T findTopExpr(Expr expr, OpType type) throws TajoException { + Preconditions.checkNotNull(expr); + Preconditions.checkNotNull(type); + + ExprFinder finder = new ExprFinder(type); + finder.visit(null, new Stack<Expr>(), expr); + + if (finder.getFoundExprs().size() == 0) { + return null; + } + return (T) finder.getFoundExprs().get(0); + } + + private static class ExprFinder extends SimpleAlgebraVisitor<Object, Expr> { + private List<Expr> list = new ArrayList<Expr>(); + private final OpType[] tofind; + private boolean topmost = false; + private boolean finished = false; + + public ExprFinder(OpType... type) { + + this.tofind = type; + } + + public ExprFinder(OpType[] type, boolean topmost) { + this(type); + this.topmost = topmost; + } + + @Override + public Expr visit(Object ctx, Stack<Expr> stack, Expr expr) throws TajoException { + if (!finished) { + for (OpType type : tofind) { + if (expr.getType() == type) { + list.add(expr); + } + if (topmost && list.size() > 0) { + finished = true; + } + } + } + return super.visit(ctx, stack, expr); + } + + public List<Expr> getFoundExprs() { + return list; + } + + } + + public static Expr[] toConjunctiveNormalFormArray(Expr expr) { + List<Expr> list = new ArrayList<Expr>(); + toConjunctiveNormalFormArrayRecursive(expr, list); + return list.toArray(new Expr[list.size()]); + } + + private static void toConjunctiveNormalFormArrayRecursive(Expr node, List<Expr> found) { — End diff – Sorry, my mistake.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r40102943

          — Diff: tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/FileTablespace.java —
          @@ -343,7 +343,7 @@ public boolean accept(Path path) {

          FileStatus[] matches = fs.globStatus(p, inputFilter);
          if (matches == null)

          { - errors.add(new IOException("Input path does not exist: " + p)); + LOG.warn("Input path does not exist: " + p); }

          else if (matches.length == 0) {
          errors.add(new IOException("Input Pattern " + p + " matches 0 files"));
          — End diff –

          This should also not be an error.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r40102943 — Diff: tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/FileTablespace.java — @@ -343,7 +343,7 @@ public boolean accept(Path path) { FileStatus[] matches = fs.globStatus(p, inputFilter); if (matches == null) { - errors.add(new IOException("Input path does not exist: " + p)); + LOG.warn("Input path does not exist: " + p); } else if (matches.length == 0) { errors.add(new IOException("Input Pattern " + p + " matches 0 files")); — End diff – This should also not be an error.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r40103478

          — Diff: tajo-common/src/main/java/org/apache/tajo/exception/PartitionNotFoundException.java —
          @@ -0,0 +1,35 @@
          +/**
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +
          +package org.apache.tajo.exception;
          +
          +import org.apache.tajo.error.Errors.ResultCode;
          +import org.apache.tajo.rpc.protocolrecords.PrimitiveProtos.ReturnState;
          +
          +public class PartitionNotFoundException extends TajoException {
          — End diff –

          This exception is duplicated with ```UndefinedPartitionException```.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r40103478 — Diff: tajo-common/src/main/java/org/apache/tajo/exception/PartitionNotFoundException.java — @@ -0,0 +1,35 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.tajo.exception; + +import org.apache.tajo.error.Errors.ResultCode; +import org.apache.tajo.rpc.protocolrecords.PrimitiveProtos.ReturnState; + +public class PartitionNotFoundException extends TajoException { — End diff – This exception is duplicated with ```UndefinedPartitionException```.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r40103698

          — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java —
          @@ -2228,6 +2238,260 @@ private void setPartitionKeys(int pid, PartitionDescProto.Builder partitionDesc)
          return partitions;
          }

          + /**
          + * Check if list of partitions exist on catalog.
          + *
          + *
          + * @param databaseId
          + * @param tableId
          + * @return
          + */
          + public boolean existPartitionsOnCatalog(int tableId) {
          + Connection conn = null;
          + ResultSet res = null;
          + PreparedStatement pstmt = null;
          + boolean result = false;
          +
          + try {
          + String sql = "SELECT * FROM " + TB_PARTTIONS +" WHERE " + COL_TABLES_PK + " = ? ";
          + if (LOG.isDebugEnabled())

          { + LOG.debug(sql); + }

          +
          + conn = getConnection();
          + pstmt = conn.prepareStatement(sql);
          + pstmt.setInt(1, tableId);
          + res = pstmt.executeQuery();
          +
          + if (res.next())

          { + result = true; + }

          + } catch (SQLException se)

          { + throw new TajoInternalError(se); + }

          finally

          { + CatalogUtil.closeQuietly(pstmt, res); + }

          + return result;
          + }
          +
          + @Override
          + public List<PartitionDescProto> getPartitionsByFilter(PartitionsByFilterProto request)

          { + throw new TajoRuntimeException(new UnsupportedException()); + }

          +
          + @Override
          + public List<PartitionDescProto> getPartitionsByAlgebra(PartitionsByAlgebraProto request) throws
          + UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException,
          + PartitionNotFoundException, UnsupportedException {
          + Connection conn = null;
          + PreparedStatement pstmt = null;
          + ResultSet res = null;
          + int currentIndex = 1;
          + String selectStatement = null;
          + Pair<String, List<PartitionFilterSet>> pair = null;
          +
          + List<PartitionDescProto> partitions = TUtil.newList();
          + List<PartitionFilterSet> filterSets = null;
          +
          + int databaseId = getDatabaseId(request.getDatabaseName());
          + int tableId = getTableId(databaseId, request.getDatabaseName(), request.getTableName());
          + if (!existPartitionMethod(request.getDatabaseName(), request.getTableName()))

          { + throw new UndefinedPartitionMethodException(request.getTableName()); + }

          +
          + if (!existPartitionsOnCatalog(tableId)) {
          + throw new PartitionNotFoundException(request.getTableName());
          — End diff –

          Why does this method still throw ```PartitionNotFoundException```? Do you have any reason?

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r40103698 — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java — @@ -2228,6 +2238,260 @@ private void setPartitionKeys(int pid, PartitionDescProto.Builder partitionDesc) return partitions; } + /** + * Check if list of partitions exist on catalog. + * + * + * @param databaseId + * @param tableId + * @return + */ + public boolean existPartitionsOnCatalog(int tableId) { + Connection conn = null; + ResultSet res = null; + PreparedStatement pstmt = null; + boolean result = false; + + try { + String sql = "SELECT * FROM " + TB_PARTTIONS +" WHERE " + COL_TABLES_PK + " = ? "; + if (LOG.isDebugEnabled()) { + LOG.debug(sql); + } + + conn = getConnection(); + pstmt = conn.prepareStatement(sql); + pstmt.setInt(1, tableId); + res = pstmt.executeQuery(); + + if (res.next()) { + result = true; + } + } catch (SQLException se) { + throw new TajoInternalError(se); + } finally { + CatalogUtil.closeQuietly(pstmt, res); + } + return result; + } + + @Override + public List<PartitionDescProto> getPartitionsByFilter(PartitionsByFilterProto request) { + throw new TajoRuntimeException(new UnsupportedException()); + } + + @Override + public List<PartitionDescProto> getPartitionsByAlgebra(PartitionsByAlgebraProto request) throws + UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException, + PartitionNotFoundException, UnsupportedException { + Connection conn = null; + PreparedStatement pstmt = null; + ResultSet res = null; + int currentIndex = 1; + String selectStatement = null; + Pair<String, List<PartitionFilterSet>> pair = null; + + List<PartitionDescProto> partitions = TUtil.newList(); + List<PartitionFilterSet> filterSets = null; + + int databaseId = getDatabaseId(request.getDatabaseName()); + int tableId = getTableId(databaseId, request.getDatabaseName(), request.getTableName()); + if (!existPartitionMethod(request.getDatabaseName(), request.getTableName())) { + throw new UndefinedPartitionMethodException(request.getTableName()); + } + + if (!existPartitionsOnCatalog(tableId)) { + throw new PartitionNotFoundException(request.getTableName()); — End diff – Why does this method still throw ```PartitionNotFoundException```? Do you have any reason?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user blrunner commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r40165175

          — Diff: tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/FileTablespace.java —
          @@ -343,7 +343,7 @@ public boolean accept(Path path) {

          FileStatus[] matches = fs.globStatus(p, inputFilter);
          if (matches == null)

          { - errors.add(new IOException("Input path does not exist: " + p)); + LOG.warn("Input path does not exist: " + p); }

          else if (matches.length == 0) {
          errors.add(new IOException("Input Pattern " + p + " matches 0 files"));
          — End diff –

          If so, how about update ```LOG.warn``` to ```LOG.info```?

          Show
          githubbot ASF GitHub Bot added a comment - Github user blrunner commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r40165175 — Diff: tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/FileTablespace.java — @@ -343,7 +343,7 @@ public boolean accept(Path path) { FileStatus[] matches = fs.globStatus(p, inputFilter); if (matches == null) { - errors.add(new IOException("Input path does not exist: " + p)); + LOG.warn("Input path does not exist: " + p); } else if (matches.length == 0) { errors.add(new IOException("Input Pattern " + p + " matches 0 files")); — End diff – If so, how about update ```LOG.warn``` to ```LOG.info```?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user blrunner commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r40165541

          — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java —
          @@ -2228,6 +2238,260 @@ private void setPartitionKeys(int pid, PartitionDescProto.Builder partitionDesc)
          return partitions;
          }

          + /**
          + * Check if list of partitions exist on catalog.
          + *
          + *
          + * @param databaseId
          + * @param tableId
          + * @return
          + */
          + public boolean existPartitionsOnCatalog(int tableId) {
          + Connection conn = null;
          + ResultSet res = null;
          + PreparedStatement pstmt = null;
          + boolean result = false;
          +
          + try {
          + String sql = "SELECT * FROM " + TB_PARTTIONS +" WHERE " + COL_TABLES_PK + " = ? ";
          + if (LOG.isDebugEnabled())

          { + LOG.debug(sql); + }

          +
          + conn = getConnection();
          + pstmt = conn.prepareStatement(sql);
          + pstmt.setInt(1, tableId);
          + res = pstmt.executeQuery();
          +
          + if (res.next())

          { + result = true; + }

          + } catch (SQLException se)

          { + throw new TajoInternalError(se); + }

          finally

          { + CatalogUtil.closeQuietly(pstmt, res); + }

          + return result;
          + }
          +
          + @Override
          + public List<PartitionDescProto> getPartitionsByFilter(PartitionsByFilterProto request)

          { + throw new TajoRuntimeException(new UnsupportedException()); + }

          +
          + @Override
          + public List<PartitionDescProto> getPartitionsByAlgebra(PartitionsByAlgebraProto request) throws
          + UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException,
          + PartitionNotFoundException, UnsupportedException {
          + Connection conn = null;
          + PreparedStatement pstmt = null;
          + ResultSet res = null;
          + int currentIndex = 1;
          + String selectStatement = null;
          + Pair<String, List<PartitionFilterSet>> pair = null;
          +
          + List<PartitionDescProto> partitions = TUtil.newList();
          + List<PartitionFilterSet> filterSets = null;
          +
          + int databaseId = getDatabaseId(request.getDatabaseName());
          + int tableId = getTableId(databaseId, request.getDatabaseName(), request.getTableName());
          + if (!existPartitionMethod(request.getDatabaseName(), request.getTableName()))

          { + throw new UndefinedPartitionMethodException(request.getTableName()); + }

          +
          + if (!existPartitionsOnCatalog(tableId)) {
          + throw new PartitionNotFoundException(request.getTableName());
          — End diff –

          ```CatalogStore::getPartitionsOfTable``` can returns empty list instead of ```PartitionNotFoundException```. But if ```CatalogStore::getPartitionsByAlgebra``` doesn't return ```PartitionNotFoundException```, ```PartitionedTableRewriter``` might not check in case of hdfs listing need.

          Show
          githubbot ASF GitHub Bot added a comment - Github user blrunner commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r40165541 — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java — @@ -2228,6 +2238,260 @@ private void setPartitionKeys(int pid, PartitionDescProto.Builder partitionDesc) return partitions; } + /** + * Check if list of partitions exist on catalog. + * + * + * @param databaseId + * @param tableId + * @return + */ + public boolean existPartitionsOnCatalog(int tableId) { + Connection conn = null; + ResultSet res = null; + PreparedStatement pstmt = null; + boolean result = false; + + try { + String sql = "SELECT * FROM " + TB_PARTTIONS +" WHERE " + COL_TABLES_PK + " = ? "; + if (LOG.isDebugEnabled()) { + LOG.debug(sql); + } + + conn = getConnection(); + pstmt = conn.prepareStatement(sql); + pstmt.setInt(1, tableId); + res = pstmt.executeQuery(); + + if (res.next()) { + result = true; + } + } catch (SQLException se) { + throw new TajoInternalError(se); + } finally { + CatalogUtil.closeQuietly(pstmt, res); + } + return result; + } + + @Override + public List<PartitionDescProto> getPartitionsByFilter(PartitionsByFilterProto request) { + throw new TajoRuntimeException(new UnsupportedException()); + } + + @Override + public List<PartitionDescProto> getPartitionsByAlgebra(PartitionsByAlgebraProto request) throws + UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException, + PartitionNotFoundException, UnsupportedException { + Connection conn = null; + PreparedStatement pstmt = null; + ResultSet res = null; + int currentIndex = 1; + String selectStatement = null; + Pair<String, List<PartitionFilterSet>> pair = null; + + List<PartitionDescProto> partitions = TUtil.newList(); + List<PartitionFilterSet> filterSets = null; + + int databaseId = getDatabaseId(request.getDatabaseName()); + int tableId = getTableId(databaseId, request.getDatabaseName(), request.getTableName()); + if (!existPartitionMethod(request.getDatabaseName(), request.getTableName())) { + throw new UndefinedPartitionMethodException(request.getTableName()); + } + + if (!existPartitionsOnCatalog(tableId)) { + throw new PartitionNotFoundException(request.getTableName()); — End diff – ```CatalogStore::getPartitionsOfTable``` can returns empty list instead of ```PartitionNotFoundException```. But if ```CatalogStore::getPartitionsByAlgebra``` doesn't return ```PartitionNotFoundException```, ```PartitionedTableRewriter``` might not check in case of hdfs listing need.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user blrunner commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r40165636

          — Diff: tajo-common/src/main/java/org/apache/tajo/exception/PartitionNotFoundException.java —
          @@ -0,0 +1,35 @@
          +/**
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +
          +package org.apache.tajo.exception;
          +
          +import org.apache.tajo.error.Errors.ResultCode;
          +import org.apache.tajo.rpc.protocolrecords.PrimitiveProtos.ReturnState;
          +
          +public class PartitionNotFoundException extends TajoException {
          — End diff –

          ```UndefinedPartitionException``` means that a specified partition doesn't exist on catalog. And ```PartitionNotFoundException`` means that there are no partitions on catalog.

          Show
          githubbot ASF GitHub Bot added a comment - Github user blrunner commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r40165636 — Diff: tajo-common/src/main/java/org/apache/tajo/exception/PartitionNotFoundException.java — @@ -0,0 +1,35 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.tajo.exception; + +import org.apache.tajo.error.Errors.ResultCode; +import org.apache.tajo.rpc.protocolrecords.PrimitiveProtos.ReturnState; + +public class PartitionNotFoundException extends TajoException { — End diff – ```UndefinedPartitionException``` means that a specified partition doesn't exist on catalog. And ```PartitionNotFoundException`` means that there are no partitions on catalog.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r40166228

          — Diff: tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/FileTablespace.java —
          @@ -343,7 +343,7 @@ public boolean accept(Path path) {

          FileStatus[] matches = fs.globStatus(p, inputFilter);
          if (matches == null)

          { - errors.add(new IOException("Input path does not exist: " + p)); + LOG.warn("Input path does not exist: " + p); }

          else if (matches.length == 0) {
          errors.add(new IOException("Input Pattern " + p + " matches 0 files"));
          — End diff –

          Would you share any reason?
          I think that users can easily recognize this case even though it isn't regarded as invalid.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r40166228 — Diff: tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/FileTablespace.java — @@ -343,7 +343,7 @@ public boolean accept(Path path) { FileStatus[] matches = fs.globStatus(p, inputFilter); if (matches == null) { - errors.add(new IOException("Input path does not exist: " + p)); + LOG.warn("Input path does not exist: " + p); } else if (matches.length == 0) { errors.add(new IOException("Input Pattern " + p + " matches 0 files")); — End diff – Would you share any reason? I think that users can easily recognize this case even though it isn't regarded as invalid.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r40166241

          — Diff: tajo-common/src/main/java/org/apache/tajo/exception/PartitionNotFoundException.java —
          @@ -0,0 +1,35 @@
          +/**
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +
          +package org.apache.tajo.exception;
          +
          +import org.apache.tajo.error.Errors.ResultCode;
          +import org.apache.tajo.rpc.protocolrecords.PrimitiveProtos.ReturnState;
          +
          +public class PartitionNotFoundException extends TajoException {
          — End diff –

          If so, I think that this exception is not necessary.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r40166241 — Diff: tajo-common/src/main/java/org/apache/tajo/exception/PartitionNotFoundException.java — @@ -0,0 +1,35 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.tajo.exception; + +import org.apache.tajo.error.Errors.ResultCode; +import org.apache.tajo.rpc.protocolrecords.PrimitiveProtos.ReturnState; + +public class PartitionNotFoundException extends TajoException { — End diff – If so, I think that this exception is not necessary.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r40166544

          — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java —
          @@ -2228,6 +2238,260 @@ private void setPartitionKeys(int pid, PartitionDescProto.Builder partitionDesc)
          return partitions;
          }

          + /**
          + * Check if list of partitions exist on catalog.
          + *
          + *
          + * @param databaseId
          + * @param tableId
          + * @return
          + */
          + public boolean existPartitionsOnCatalog(int tableId) {
          + Connection conn = null;
          + ResultSet res = null;
          + PreparedStatement pstmt = null;
          + boolean result = false;
          +
          + try {
          + String sql = "SELECT * FROM " + TB_PARTTIONS +" WHERE " + COL_TABLES_PK + " = ? ";
          + if (LOG.isDebugEnabled())

          { + LOG.debug(sql); + }

          +
          + conn = getConnection();
          + pstmt = conn.prepareStatement(sql);
          + pstmt.setInt(1, tableId);
          + res = pstmt.executeQuery();
          +
          + if (res.next())

          { + result = true; + }

          + } catch (SQLException se)

          { + throw new TajoInternalError(se); + }

          finally

          { + CatalogUtil.closeQuietly(pstmt, res); + }

          + return result;
          + }
          +
          + @Override
          + public List<PartitionDescProto> getPartitionsByFilter(PartitionsByFilterProto request)

          { + throw new TajoRuntimeException(new UnsupportedException()); + }

          +
          + @Override
          + public List<PartitionDescProto> getPartitionsByAlgebra(PartitionsByAlgebraProto request) throws
          + UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException,
          + PartitionNotFoundException, UnsupportedException {
          + Connection conn = null;
          + PreparedStatement pstmt = null;
          + ResultSet res = null;
          + int currentIndex = 1;
          + String selectStatement = null;
          + Pair<String, List<PartitionFilterSet>> pair = null;
          +
          + List<PartitionDescProto> partitions = TUtil.newList();
          + List<PartitionFilterSet> filterSets = null;
          +
          + int databaseId = getDatabaseId(request.getDatabaseName());
          + int tableId = getTableId(databaseId, request.getDatabaseName(), request.getTableName());
          + if (!existPartitionMethod(request.getDatabaseName(), request.getTableName()))

          { + throw new UndefinedPartitionMethodException(request.getTableName()); + }

          +
          + if (!existPartitionsOnCatalog(tableId)) {
          + throw new PartitionNotFoundException(request.getTableName());
          — End diff –

          It's difficult for me to understand why those implementations are different.
          The exception means an invalid situation. So, basically, exceptions should be handled as error cases. However, the case you explained above is not an error case.

          In addition, this exception is never exposed to users, but used only in PartitionedTableRewriter. I think this can be replaced with a better routine.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r40166544 — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java — @@ -2228,6 +2238,260 @@ private void setPartitionKeys(int pid, PartitionDescProto.Builder partitionDesc) return partitions; } + /** + * Check if list of partitions exist on catalog. + * + * + * @param databaseId + * @param tableId + * @return + */ + public boolean existPartitionsOnCatalog(int tableId) { + Connection conn = null; + ResultSet res = null; + PreparedStatement pstmt = null; + boolean result = false; + + try { + String sql = "SELECT * FROM " + TB_PARTTIONS +" WHERE " + COL_TABLES_PK + " = ? "; + if (LOG.isDebugEnabled()) { + LOG.debug(sql); + } + + conn = getConnection(); + pstmt = conn.prepareStatement(sql); + pstmt.setInt(1, tableId); + res = pstmt.executeQuery(); + + if (res.next()) { + result = true; + } + } catch (SQLException se) { + throw new TajoInternalError(se); + } finally { + CatalogUtil.closeQuietly(pstmt, res); + } + return result; + } + + @Override + public List<PartitionDescProto> getPartitionsByFilter(PartitionsByFilterProto request) { + throw new TajoRuntimeException(new UnsupportedException()); + } + + @Override + public List<PartitionDescProto> getPartitionsByAlgebra(PartitionsByAlgebraProto request) throws + UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException, + PartitionNotFoundException, UnsupportedException { + Connection conn = null; + PreparedStatement pstmt = null; + ResultSet res = null; + int currentIndex = 1; + String selectStatement = null; + Pair<String, List<PartitionFilterSet>> pair = null; + + List<PartitionDescProto> partitions = TUtil.newList(); + List<PartitionFilterSet> filterSets = null; + + int databaseId = getDatabaseId(request.getDatabaseName()); + int tableId = getTableId(databaseId, request.getDatabaseName(), request.getTableName()); + if (!existPartitionMethod(request.getDatabaseName(), request.getTableName())) { + throw new UndefinedPartitionMethodException(request.getTableName()); + } + + if (!existPartitionsOnCatalog(tableId)) { + throw new PartitionNotFoundException(request.getTableName()); — End diff – It's difficult for me to understand why those implementations are different. The exception means an invalid situation. So, basically, exceptions should be handled as error cases. However, the case you explained above is not an error case. In addition, this exception is never exposed to users, but used only in PartitionedTableRewriter. I think this can be replaced with a better routine.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user blrunner commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r40167412

          — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java —
          @@ -2228,6 +2238,260 @@ private void setPartitionKeys(int pid, PartitionDescProto.Builder partitionDesc)
          return partitions;
          }

          + /**
          + * Check if list of partitions exist on catalog.
          + *
          + *
          + * @param databaseId
          + * @param tableId
          + * @return
          + */
          + public boolean existPartitionsOnCatalog(int tableId) {
          + Connection conn = null;
          + ResultSet res = null;
          + PreparedStatement pstmt = null;
          + boolean result = false;
          +
          + try {
          + String sql = "SELECT * FROM " + TB_PARTTIONS +" WHERE " + COL_TABLES_PK + " = ? ";
          + if (LOG.isDebugEnabled())

          { + LOG.debug(sql); + }

          +
          + conn = getConnection();
          + pstmt = conn.prepareStatement(sql);
          + pstmt.setInt(1, tableId);
          + res = pstmt.executeQuery();
          +
          + if (res.next())

          { + result = true; + }

          + } catch (SQLException se)

          { + throw new TajoInternalError(se); + }

          finally

          { + CatalogUtil.closeQuietly(pstmt, res); + }

          + return result;
          + }
          +
          + @Override
          + public List<PartitionDescProto> getPartitionsByFilter(PartitionsByFilterProto request)

          { + throw new TajoRuntimeException(new UnsupportedException()); + }

          +
          + @Override
          + public List<PartitionDescProto> getPartitionsByAlgebra(PartitionsByAlgebraProto request) throws
          + UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException,
          + PartitionNotFoundException, UnsupportedException {
          + Connection conn = null;
          + PreparedStatement pstmt = null;
          + ResultSet res = null;
          + int currentIndex = 1;
          + String selectStatement = null;
          + Pair<String, List<PartitionFilterSet>> pair = null;
          +
          + List<PartitionDescProto> partitions = TUtil.newList();
          + List<PartitionFilterSet> filterSets = null;
          +
          + int databaseId = getDatabaseId(request.getDatabaseName());
          + int tableId = getTableId(databaseId, request.getDatabaseName(), request.getTableName());
          + if (!existPartitionMethod(request.getDatabaseName(), request.getTableName()))

          { + throw new UndefinedPartitionMethodException(request.getTableName()); + }

          +
          + if (!existPartitionsOnCatalog(tableId)) {
          + throw new PartitionNotFoundException(request.getTableName());
          — End diff –

          These implementations are just for work loads. If users recover their partitions using https://github.com/apache/tajo/pull/626, this exception is never exposed to users. If we notice this point to users and decide to ignore above situation, we can remove these implementations.

          Show
          githubbot ASF GitHub Bot added a comment - Github user blrunner commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r40167412 — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java — @@ -2228,6 +2238,260 @@ private void setPartitionKeys(int pid, PartitionDescProto.Builder partitionDesc) return partitions; } + /** + * Check if list of partitions exist on catalog. + * + * + * @param databaseId + * @param tableId + * @return + */ + public boolean existPartitionsOnCatalog(int tableId) { + Connection conn = null; + ResultSet res = null; + PreparedStatement pstmt = null; + boolean result = false; + + try { + String sql = "SELECT * FROM " + TB_PARTTIONS +" WHERE " + COL_TABLES_PK + " = ? "; + if (LOG.isDebugEnabled()) { + LOG.debug(sql); + } + + conn = getConnection(); + pstmt = conn.prepareStatement(sql); + pstmt.setInt(1, tableId); + res = pstmt.executeQuery(); + + if (res.next()) { + result = true; + } + } catch (SQLException se) { + throw new TajoInternalError(se); + } finally { + CatalogUtil.closeQuietly(pstmt, res); + } + return result; + } + + @Override + public List<PartitionDescProto> getPartitionsByFilter(PartitionsByFilterProto request) { + throw new TajoRuntimeException(new UnsupportedException()); + } + + @Override + public List<PartitionDescProto> getPartitionsByAlgebra(PartitionsByAlgebraProto request) throws + UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException, + PartitionNotFoundException, UnsupportedException { + Connection conn = null; + PreparedStatement pstmt = null; + ResultSet res = null; + int currentIndex = 1; + String selectStatement = null; + Pair<String, List<PartitionFilterSet>> pair = null; + + List<PartitionDescProto> partitions = TUtil.newList(); + List<PartitionFilterSet> filterSets = null; + + int databaseId = getDatabaseId(request.getDatabaseName()); + int tableId = getTableId(databaseId, request.getDatabaseName(), request.getTableName()); + if (!existPartitionMethod(request.getDatabaseName(), request.getTableName())) { + throw new UndefinedPartitionMethodException(request.getTableName()); + } + + if (!existPartitionsOnCatalog(tableId)) { + throw new PartitionNotFoundException(request.getTableName()); — End diff – These implementations are just for work loads. If users recover their partitions using https://github.com/apache/tajo/pull/626 , this exception is never exposed to users. If we notice this point to users and decide to ignore above situation, we can remove these implementations.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user blrunner commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r40167828

          — Diff: tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/FileTablespace.java —
          @@ -343,7 +343,7 @@ public boolean accept(Path path) {

          FileStatus[] matches = fs.globStatus(p, inputFilter);
          if (matches == null)

          { - errors.add(new IOException("Input path does not exist: " + p)); + LOG.warn("Input path does not exist: " + p); }

          else if (matches.length == 0) {
          errors.add(new IOException("Input Pattern " + p + " matches 0 files"));
          — End diff –

          If users can easily recognize this case, it doesn't matter.

          Show
          githubbot ASF GitHub Bot added a comment - Github user blrunner commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r40167828 — Diff: tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/FileTablespace.java — @@ -343,7 +343,7 @@ public boolean accept(Path path) { FileStatus[] matches = fs.globStatus(p, inputFilter); if (matches == null) { - errors.add(new IOException("Input path does not exist: " + p)); + LOG.warn("Input path does not exist: " + p); } else if (matches.length == 0) { errors.add(new IOException("Input Pattern " + p + " matches 0 files")); — End diff – If users can easily recognize this case, it doesn't matter.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user blrunner commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r40168039

          — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java —
          @@ -2228,6 +2244,257 @@ private void setPartitionKeys(int pid, PartitionDescProto.Builder partitionDesc)
          return partitions;
          }

          + /**
          + * Check if list of partitions exist on catalog.
          + *
          + *
          + * @param databaseId
          + * @param tableId
          + * @return
          + */
          + public boolean existPartitionsOnCatalog(int tableId) {
          + Connection conn = null;
          + ResultSet res = null;
          + PreparedStatement pstmt = null;
          + boolean result = false;
          +
          + try {
          + String sql = "SELECT COUNT CNT FROM "
          + + TB_PARTTIONS +" WHERE " + COL_TABLES_PK + " = ? ";
          +
          + if (LOG.isDebugEnabled())

          { + LOG.debug(sql); + }

          +
          + conn = getConnection();
          + pstmt = conn.prepareStatement(sql);
          + pstmt.setInt(1, tableId);
          + res = pstmt.executeQuery();
          +
          + if (res.next()) {
          + if (res.getInt("CNT") > 0)

          { + result = true; + }

          + }
          + } catch (SQLException se)

          { + throw new TajoInternalError(se); + }

          finally

          { + CatalogUtil.closeQuietly(pstmt, res); + }

          + return result;
          + }
          +
          + @Override
          + public List<PartitionDescProto> getPartitionsByFilter(PartitionsByFilterProto request) {
          + throw new TajoRuntimeException(new UnsupportedException());
          — End diff –

          Thank you for your understanding. Could you check the comments on CatalogStore?

          Show
          githubbot ASF GitHub Bot added a comment - Github user blrunner commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r40168039 — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java — @@ -2228,6 +2244,257 @@ private void setPartitionKeys(int pid, PartitionDescProto.Builder partitionDesc) return partitions; } + /** + * Check if list of partitions exist on catalog. + * + * + * @param databaseId + * @param tableId + * @return + */ + public boolean existPartitionsOnCatalog(int tableId) { + Connection conn = null; + ResultSet res = null; + PreparedStatement pstmt = null; + boolean result = false; + + try { + String sql = "SELECT COUNT CNT FROM " + + TB_PARTTIONS +" WHERE " + COL_TABLES_PK + " = ? "; + + if (LOG.isDebugEnabled()) { + LOG.debug(sql); + } + + conn = getConnection(); + pstmt = conn.prepareStatement(sql); + pstmt.setInt(1, tableId); + res = pstmt.executeQuery(); + + if (res.next()) { + if (res.getInt("CNT") > 0) { + result = true; + } + } + } catch (SQLException se) { + throw new TajoInternalError(se); + } finally { + CatalogUtil.closeQuietly(pstmt, res); + } + return result; + } + + @Override + public List<PartitionDescProto> getPartitionsByFilter(PartitionsByFilterProto request) { + throw new TajoRuntimeException(new UnsupportedException()); — End diff – Thank you for your understanding. Could you check the comments on CatalogStore?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user blrunner commented on the pull request:

          https://github.com/apache/tajo/pull/772#issuecomment-142506156

          Thanks @jihoonson

          I've reflected your comments and I'm going to create jia issues about ```CatalogStore::getPartitionsByFilter``` and performance of the select statement for getting partitions after finishing this PR.

          Show
          githubbot ASF GitHub Bot added a comment - Github user blrunner commented on the pull request: https://github.com/apache/tajo/pull/772#issuecomment-142506156 Thanks @jihoonson I've reflected your comments and I'm going to create jia issues about ```CatalogStore::getPartitionsByFilter``` and performance of the select statement for getting partitions after finishing this PR.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r40201582

          — Diff: tajo-cli/src/main/java/org/apache/tajo/cli/tools/TajoDump.java —
          @@ -208,6 +208,7 @@ private static void dumpDatabase(TajoClient client, String databaseName, PrintWr
          }
          writer.write("\n\n");
          } catch (Exception e) {
          + e.printStackTrace();
          — End diff –

          Please remove this line.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r40201582 — Diff: tajo-cli/src/main/java/org/apache/tajo/cli/tools/TajoDump.java — @@ -208,6 +208,7 @@ private static void dumpDatabase(TajoClient client, String databaseName, PrintWr } writer.write("\n\n"); } catch (Exception e) { + e.printStackTrace(); — End diff – Please remove this line.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r40201715

          — Diff: tajo-common/src/main/java/org/apache/tajo/exception/UndefinedPartitionsException.java —
          @@ -0,0 +1,35 @@
          +/**
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +
          +package org.apache.tajo.exception;
          +
          +import org.apache.tajo.error.Errors.ResultCode;
          +import org.apache.tajo.rpc.protocolrecords.PrimitiveProtos.ReturnState;
          +
          +public class UndefinedPartitionsException extends TajoException {
          — End diff –

          This exception is not used.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r40201715 — Diff: tajo-common/src/main/java/org/apache/tajo/exception/UndefinedPartitionsException.java — @@ -0,0 +1,35 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.tajo.exception; + +import org.apache.tajo.error.Errors.ResultCode; +import org.apache.tajo.rpc.protocolrecords.PrimitiveProtos.ReturnState; + +public class UndefinedPartitionsException extends TajoException { — End diff – This exception is not used.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r40202591

          — Diff: tajo-plan/src/main/java/org/apache/tajo/plan/rewrite/rules/PartitionedTableRewriter.java —
          @@ -118,13 +120,76 @@ public String toString() {

          • @return
          • @throws IOException
            */
          • private Path [] findFilteredPaths(OverridableConf queryContext, Schema partitionColumns, EvalNode [] conjunctiveForms,
          • Path tablePath)
          • throws IOException {
            + private Path [] findFilteredPaths(OverridableConf queryContext, String tableName,
            + Schema partitionColumns, EvalNode [] conjunctiveForms, Path tablePath)
            + throws IOException, UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException,
            + UndefinedOperatorException, UnsupportedException {

          + Path [] filteredPaths = null;
          FileSystem fs = tablePath.getFileSystem(queryContext.getConf());
          + String [] splits = CatalogUtil.splitFQTableName(tableName);
          + List<PartitionDescProto> partitions = null;
          +
          + try {
          + if (conjunctiveForms == null) {
          + partitions = catalog.getPartitionsOfTable(splits[0], splits[1]);
          + if (partitions.isEmpty())

          { + filteredPaths = findFilteredPathsFromFileSystem(partitionColumns, conjunctiveForms, fs, tablePath); + } else { + filteredPaths = findFilteredPathsByPartitionDesc(partitions); + }
          + } else {
          + if (catalog.existPartitions(splits[0], splits[1])) { + PartitionsByAlgebraProto request = getPartitionsAlgebraProto(splits[0], splits[1], conjunctiveForms); + partitions = catalog.getPartitionsByAlgebra(request); + filteredPaths = findFilteredPathsByPartitionDesc(partitions); + } else { + filteredPaths = findFilteredPathsFromFileSystem(partitionColumns, conjunctiveForms, fs, tablePath); + }

          + }
          + } catch (UnsupportedException ue) {
          — End diff –

          IMO, this case is regarded as the full scan on partitions.
          So, the same routine with the above ```if (conjunctiveForms == null)``` clause is required here.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r40202591 — Diff: tajo-plan/src/main/java/org/apache/tajo/plan/rewrite/rules/PartitionedTableRewriter.java — @@ -118,13 +120,76 @@ public String toString() { @return @throws IOException */ private Path [] findFilteredPaths(OverridableConf queryContext, Schema partitionColumns, EvalNode [] conjunctiveForms, Path tablePath) throws IOException { + private Path [] findFilteredPaths(OverridableConf queryContext, String tableName, + Schema partitionColumns, EvalNode [] conjunctiveForms, Path tablePath) + throws IOException, UndefinedDatabaseException, UndefinedTableException, UndefinedPartitionMethodException, + UndefinedOperatorException, UnsupportedException { + Path [] filteredPaths = null; FileSystem fs = tablePath.getFileSystem(queryContext.getConf()); + String [] splits = CatalogUtil.splitFQTableName(tableName); + List<PartitionDescProto> partitions = null; + + try { + if (conjunctiveForms == null) { + partitions = catalog.getPartitionsOfTable(splits [0] , splits [1] ); + if (partitions.isEmpty()) { + filteredPaths = findFilteredPathsFromFileSystem(partitionColumns, conjunctiveForms, fs, tablePath); + } else { + filteredPaths = findFilteredPathsByPartitionDesc(partitions); + } + } else { + if (catalog.existPartitions(splits [0] , splits [1] )) { + PartitionsByAlgebraProto request = getPartitionsAlgebraProto(splits[0], splits[1], conjunctiveForms); + partitions = catalog.getPartitionsByAlgebra(request); + filteredPaths = findFilteredPathsByPartitionDesc(partitions); + } else { + filteredPaths = findFilteredPathsFromFileSystem(partitionColumns, conjunctiveForms, fs, tablePath); + } + } + } catch (UnsupportedException ue) { — End diff – IMO, this case is regarded as the full scan on partitions. So, the same routine with the above ```if (conjunctiveForms == null)``` clause is required here.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/772#discussion_r40203039

          — Diff: tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/FileTablespace.java —
          @@ -342,18 +342,18 @@ public boolean accept(Path path) {
          Path p = dirs[i];

          FileStatus[] matches = fs.globStatus(p, inputFilter);

          • if (matches == null) { - errors.add(new IOException("Input path does not exist: " + p)); - }

            else if (matches.length == 0)

            { - errors.add(new IOException("Input Pattern " + p + " matches 0 files")); - }

            else {

          • for (FileStatus globStat : matches) {
          • if (globStat.isDirectory()) {
          • for (FileStatus stat : fs.listStatus(globStat.getPath(), inputFilter)) {
          • result.add(stat);
            + if (matches != null) {
            + if (matches.length == 0) {
              • End diff –

          If ```matches``` is null or ```matches.length``` is 0, you need to print a warning message.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/772#discussion_r40203039 — Diff: tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/FileTablespace.java — @@ -342,18 +342,18 @@ public boolean accept(Path path) { Path p = dirs [i] ; FileStatus[] matches = fs.globStatus(p, inputFilter); if (matches == null) { - errors.add(new IOException("Input path does not exist: " + p)); - } else if (matches.length == 0) { - errors.add(new IOException("Input Pattern " + p + " matches 0 files")); - } else { for (FileStatus globStat : matches) { if (globStat.isDirectory()) { for (FileStatus stat : fs.listStatus(globStat.getPath(), inputFilter)) { result.add(stat); + if (matches != null) { + if (matches.length == 0) { End diff – If ```matches``` is null or ```matches.length``` is 0, you need to print a warning message.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user blrunner commented on the pull request:

          https://github.com/apache/tajo/pull/772#issuecomment-142636766

          Thanks @jihoonson .
          I've just reflected your comments. Could you check it again?

          Show
          githubbot ASF GitHub Bot added a comment - Github user blrunner commented on the pull request: https://github.com/apache/tajo/pull/772#issuecomment-142636766 Thanks @jihoonson . I've just reflected your comments. Could you check it again?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user blrunner commented on the pull request:

          https://github.com/apache/tajo/pull/772#issuecomment-142674077

          I've updated the patch as following:

          • *Allow invalid partition directory name*: When adding partition using alter statement, users can set partition path freely. But current PartitionedTableRewriter just allow well-defined partition directory which consists of column name and partition value. So if users use invalid partition directory name, they can scan their partitions. To resolve above situation, I added list of PartitionDescProto to PartitionedTableScanNode.
          Show
          githubbot ASF GitHub Bot added a comment - Github user blrunner commented on the pull request: https://github.com/apache/tajo/pull/772#issuecomment-142674077 I've updated the patch as following: * Remove ```CatalogStore::getPartitionsByFilter``` *: AbstractDBStore doesn't support this method. Also HiveCatalogStore support the method restrictedly. You can see the grammar for hive partition api to https://github.com/apache/hive/blob/master/metastore/src/java/org/apache/hadoop/hive/metastore/parser/Filter.g . But I think that we need the method for user convenience. So I'll implement at another jira issue using tajo-sql-parser module and ```CatalogStore::getPartitionsByAlgebra```. * Allow invalid partition directory name *: When adding partition using alter statement, users can set partition path freely. But current PartitionedTableRewriter just allow well-defined partition directory which consists of column name and partition value. So if users use invalid partition directory name, they can scan their partitions. To resolve above situation, I added list of PartitionDescProto to PartitionedTableScanNode.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user blrunner commented on the pull request:

          https://github.com/apache/tajo/pull/772#issuecomment-142839704

          All unit test finished successfully on my laptop. And I tested this patch successfully as following:

          • Data: TPC-H 1G
          • Query: Q1, Q3, Q4, Q5, Q8
          • Catalog : MySQLStore, HiveCatalogStore
          • Table schema:

          ```
          CREATE TABLE customer (c_custkey INT8, c_name TEXT, c_address TEXT, c_phone TEXT, c_acctbal FLOAT8, c_mktsegment TEXT, c_comment TEXT) USING TEXT PARTITION BY COLUMN(c_nationkey INT8) ;

          CREATE TABLE lineitem (l_orderkey INT8, l_partkey INT8, l_suppkey INT8, l_linenumber INT8, l_quantity FLOAT8, l_extendedprice FLOAT8, l_discount FLOAT8, l_tax FLOAT8, l_commitdate DATE, l_receiptdate DATE, l_shipinstruct TEXT, l_shipmode TEXT, l_comment TEXT) USING TEXT PARTITION BY COLUMN(l_shipdate DATE, l_returnflag TEXT, l_linestatus TEXT) ;

          CREATE TABLE nation (n_nationkey INT8, n_name TEXT, n_comment TEXT) USING TEXT PARTITION BY COLUMN(n_regionkey INT8) ;

          CREATE TABLE orders (o_orderkey INT8, o_custkey INT8, o_totalprice FLOAT8, o_clerk TEXT, o_shippriority INT4, o_comment TEXT) USING TEXT PARTITION BY COLUMN(o_orderdate DATE, o_orderstatus TEXT, o_orderpriority TEXT) ;

          CREATE TABLE part (p_partkey INT8, p_name TEXT, p_mfgr TEXT, p_brand TEXT, p_type TEXT, p_container TEXT, p_retailprice FLOAT8, p_comment TEXT) USING TEXT PARTITION BY COLUMN(p_size INT4) ;

          CREATE TABLE partsupp (ps_partkey INT8, ps_suppkey INT8, ps_availqty INT4, ps_supplycost FLOAT8, ps_comment TEXT) USING TEXT ;

          CREATE TABLE region (r_name TEXT, r_comment TEXT) USING TEXT PARTITION BY COLUMN(r_regionkey INT8);

          CREATE TABLE supplier (s_suppkey INT8, s_name TEXT, s_address TEXT, s_phone TEXT, s_acctbal FLOAT8, s_comment TEXT) USING TEXT PARTITION BY COLUMN(s_nationkey INT8);
          ```

          Show
          githubbot ASF GitHub Bot added a comment - Github user blrunner commented on the pull request: https://github.com/apache/tajo/pull/772#issuecomment-142839704 All unit test finished successfully on my laptop. And I tested this patch successfully as following: Data: TPC-H 1G Query: Q1, Q3, Q4, Q5, Q8 Catalog : MySQLStore, HiveCatalogStore Table schema: ``` CREATE TABLE customer (c_custkey INT8, c_name TEXT, c_address TEXT, c_phone TEXT, c_acctbal FLOAT8, c_mktsegment TEXT, c_comment TEXT) USING TEXT PARTITION BY COLUMN(c_nationkey INT8) ; CREATE TABLE lineitem (l_orderkey INT8, l_partkey INT8, l_suppkey INT8, l_linenumber INT8, l_quantity FLOAT8, l_extendedprice FLOAT8, l_discount FLOAT8, l_tax FLOAT8, l_commitdate DATE, l_receiptdate DATE, l_shipinstruct TEXT, l_shipmode TEXT, l_comment TEXT) USING TEXT PARTITION BY COLUMN(l_shipdate DATE, l_returnflag TEXT, l_linestatus TEXT) ; CREATE TABLE nation (n_nationkey INT8, n_name TEXT, n_comment TEXT) USING TEXT PARTITION BY COLUMN(n_regionkey INT8) ; CREATE TABLE orders (o_orderkey INT8, o_custkey INT8, o_totalprice FLOAT8, o_clerk TEXT, o_shippriority INT4, o_comment TEXT) USING TEXT PARTITION BY COLUMN(o_orderdate DATE, o_orderstatus TEXT, o_orderpriority TEXT) ; CREATE TABLE part (p_partkey INT8, p_name TEXT, p_mfgr TEXT, p_brand TEXT, p_type TEXT, p_container TEXT, p_retailprice FLOAT8, p_comment TEXT) USING TEXT PARTITION BY COLUMN(p_size INT4) ; CREATE TABLE partsupp (ps_partkey INT8, ps_suppkey INT8, ps_availqty INT4, ps_supplycost FLOAT8, ps_comment TEXT) USING TEXT ; CREATE TABLE region (r_name TEXT, r_comment TEXT) USING TEXT PARTITION BY COLUMN(r_regionkey INT8); CREATE TABLE supplier (s_suppkey INT8, s_name TEXT, s_address TEXT, s_phone TEXT, s_acctbal FLOAT8, s_comment TEXT) USING TEXT PARTITION BY COLUMN(s_nationkey INT8); ```
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on the pull request:

          https://github.com/apache/tajo/pull/772#issuecomment-142859219

          +1 the latest patch looks good to me.
          Thank you for your effort!

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on the pull request: https://github.com/apache/tajo/pull/772#issuecomment-142859219 +1 the latest patch looks good to me. Thank you for your effort!
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user blrunner commented on the pull request:

          https://github.com/apache/tajo/pull/772#issuecomment-142860543

          @jihoonson

          Thank you for your detailed review. I'll ship it.

          Show
          githubbot ASF GitHub Bot added a comment - Github user blrunner commented on the pull request: https://github.com/apache/tajo/pull/772#issuecomment-142860543 @jihoonson Thank you for your detailed review. I'll ship it.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/tajo/pull/772

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/tajo/pull/772
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Tajo-master-CODEGEN-build #538 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/538/)
          TAJO-1493: Make partition pruning based on catalog informations. (blrunner: rev b5aacb917b71ae610aed9d0aa4eb76888ccc804e)

          • tajo-catalog/tajo-catalog-server/src/main/resources/schemas/mariadb/mariadb.xml
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java
          • tajo-core/src/main/java/org/apache/tajo/master/TajoMasterClientService.java
          • tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestTablePartitions.java
          • tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto
          • CHANGES
          • tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/CatalogTestingUtil.java
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogService.java
          • tajo-core/src/main/java/org/apache/tajo/querymaster/Query.java
          • tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestAlterTable.java
          • tajo-catalog/tajo-catalog-client/src/main/java/org/apache/tajo/catalog/AbstractCatalogClient.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/ColPartitionStoreExec.java
          • tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/FileTablespace.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/expr/AlgebraicUtil.java
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/partition/PartitionDesc.java
          • tajo-client/src/main/java/org/apache/tajo/client/TajoClientImpl.java
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogConstants.java
          • tajo-catalog/tajo-catalog-client/src/main/proto/CatalogProtocol.proto
          • tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogStore.java
          • tajo-common/src/main/java/org/apache/tajo/exception/ReturnStateUtil.java
          • tajo-common/src/main/java/org/apache/tajo/exception/ErrorMessages.java
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/DDLBuilder.java
          • tajo-common/src/main/java/org/apache/tajo/exception/UndefinedPartitionMethodException.java
          • tajo-client/src/main/java/org/apache/tajo/client/CatalogAdminClientImpl.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/util/PartitionFilterAlgebraVisitor.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/rewrite/rules/PartitionedTableRewriter.java
          • tajo-catalog/tajo-catalog-drivers/tajo-hive/src/test/java/org/apache/tajo/catalog/store/TestHiveCatalogStore.java
          • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java
          • tajo-catalog/tajo-catalog-server/src/main/resources/schemas/oracle/oracle.xml
          • tajo-cli/src/main/java/org/apache/tajo/cli/tools/TajoDump.java
          • tajo-catalog/tajo-catalog-server/pom.xml
          • tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalogAgainstCaseSensitivity.java
          • tajo-catalog/tajo-catalog-server/src/main/resources/schemas/derby/derby.xml
          • tajo-client/src/main/java/org/apache/tajo/client/CatalogAdminClient.java
          • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java
          • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/CatalogStore.java
          • tajo-catalog/tajo-catalog-server/src/main/resources/schemas/mysql/mysql.xml
          • tajo-plan/src/main/java/org/apache/tajo/plan/util/EvalNodeToExprConverter.java
          • tajo-core/src/main/java/org/apache/tajo/master/exec/DDLExecutor.java
          • tajo-common/src/main/java/org/apache/tajo/exception/ExceptionUtil.java
          • tajo-common/src/main/proto/errors.proto
          • tajo-catalog/tajo-catalog-server/src/main/resources/schemas/postgresql/postgresql.xml
          • tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalog.java
          • tajo-core-tests/src/test/java/org/apache/tajo/engine/planner/TestEvalNodeToExprConverter.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Tajo-master-CODEGEN-build #538 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/538/ ) TAJO-1493 : Make partition pruning based on catalog informations. (blrunner: rev b5aacb917b71ae610aed9d0aa4eb76888ccc804e) tajo-catalog/tajo-catalog-server/src/main/resources/schemas/mariadb/mariadb.xml tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java tajo-core/src/main/java/org/apache/tajo/master/TajoMasterClientService.java tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestTablePartitions.java tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto CHANGES tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/CatalogTestingUtil.java tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogService.java tajo-core/src/main/java/org/apache/tajo/querymaster/Query.java tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestAlterTable.java tajo-catalog/tajo-catalog-client/src/main/java/org/apache/tajo/catalog/AbstractCatalogClient.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/ColPartitionStoreExec.java tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/FileTablespace.java tajo-plan/src/main/java/org/apache/tajo/plan/expr/AlgebraicUtil.java tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/partition/PartitionDesc.java tajo-client/src/main/java/org/apache/tajo/client/TajoClientImpl.java tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogConstants.java tajo-catalog/tajo-catalog-client/src/main/proto/CatalogProtocol.proto tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogStore.java tajo-common/src/main/java/org/apache/tajo/exception/ReturnStateUtil.java tajo-common/src/main/java/org/apache/tajo/exception/ErrorMessages.java tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/DDLBuilder.java tajo-common/src/main/java/org/apache/tajo/exception/UndefinedPartitionMethodException.java tajo-client/src/main/java/org/apache/tajo/client/CatalogAdminClientImpl.java tajo-plan/src/main/java/org/apache/tajo/plan/util/PartitionFilterAlgebraVisitor.java tajo-plan/src/main/java/org/apache/tajo/plan/rewrite/rules/PartitionedTableRewriter.java tajo-catalog/tajo-catalog-drivers/tajo-hive/src/test/java/org/apache/tajo/catalog/store/TestHiveCatalogStore.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java tajo-catalog/tajo-catalog-server/src/main/resources/schemas/oracle/oracle.xml tajo-cli/src/main/java/org/apache/tajo/cli/tools/TajoDump.java tajo-catalog/tajo-catalog-server/pom.xml tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalogAgainstCaseSensitivity.java tajo-catalog/tajo-catalog-server/src/main/resources/schemas/derby/derby.xml tajo-client/src/main/java/org/apache/tajo/client/CatalogAdminClient.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/CatalogStore.java tajo-catalog/tajo-catalog-server/src/main/resources/schemas/mysql/mysql.xml tajo-plan/src/main/java/org/apache/tajo/plan/util/EvalNodeToExprConverter.java tajo-core/src/main/java/org/apache/tajo/master/exec/DDLExecutor.java tajo-common/src/main/java/org/apache/tajo/exception/ExceptionUtil.java tajo-common/src/main/proto/errors.proto tajo-catalog/tajo-catalog-server/src/main/resources/schemas/postgresql/postgresql.xml tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalog.java tajo-core-tests/src/test/java/org/apache/tajo/engine/planner/TestEvalNodeToExprConverter.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Tajo-master-build #905 (See https://builds.apache.org/job/Tajo-master-build/905/)
          TAJO-1493: Make partition pruning based on catalog informations. (blrunner: rev b5aacb917b71ae610aed9d0aa4eb76888ccc804e)

          • tajo-plan/src/main/java/org/apache/tajo/plan/util/EvalNodeToExprConverter.java
          • tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalog.java
          • tajo-client/src/main/java/org/apache/tajo/client/CatalogAdminClient.java
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogService.java
          • tajo-catalog/tajo-catalog-client/src/main/proto/CatalogProtocol.proto
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogConstants.java
          • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java
          • tajo-core/src/main/java/org/apache/tajo/querymaster/Query.java
          • tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto
          • tajo-client/src/main/java/org/apache/tajo/client/TajoClientImpl.java
          • CHANGES
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/partition/PartitionDesc.java
          • tajo-catalog/tajo-catalog-server/src/main/resources/schemas/derby/derby.xml
          • tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/CatalogTestingUtil.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/expr/AlgebraicUtil.java
          • tajo-cli/src/main/java/org/apache/tajo/cli/tools/TajoDump.java
          • tajo-common/src/main/java/org/apache/tajo/exception/UndefinedPartitionMethodException.java
          • tajo-catalog/tajo-catalog-client/src/main/java/org/apache/tajo/catalog/AbstractCatalogClient.java
          • tajo-core-tests/src/test/java/org/apache/tajo/engine/planner/TestEvalNodeToExprConverter.java
          • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/CatalogStore.java
          • tajo-catalog/tajo-catalog-server/src/main/resources/schemas/oracle/oracle.xml
          • tajo-common/src/main/java/org/apache/tajo/exception/ExceptionUtil.java
          • tajo-catalog/tajo-catalog-server/pom.xml
          • tajo-plan/src/main/java/org/apache/tajo/plan/util/PartitionFilterAlgebraVisitor.java
          • tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestAlterTable.java
          • tajo-common/src/main/proto/errors.proto
          • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java
          • tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestTablePartitions.java
          • tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalogAgainstCaseSensitivity.java
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/DDLBuilder.java
          • tajo-client/src/main/java/org/apache/tajo/client/CatalogAdminClientImpl.java
          • tajo-catalog/tajo-catalog-server/src/main/resources/schemas/mysql/mysql.xml
          • tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogStore.java
          • tajo-catalog/tajo-catalog-drivers/tajo-hive/src/test/java/org/apache/tajo/catalog/store/TestHiveCatalogStore.java
          • tajo-core/src/main/java/org/apache/tajo/master/TajoMasterClientService.java
          • tajo-common/src/main/java/org/apache/tajo/exception/ReturnStateUtil.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/rewrite/rules/PartitionedTableRewriter.java
          • tajo-common/src/main/java/org/apache/tajo/exception/ErrorMessages.java
          • tajo-core/src/main/java/org/apache/tajo/master/exec/DDLExecutor.java
          • tajo-catalog/tajo-catalog-server/src/main/resources/schemas/postgresql/postgresql.xml
          • tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/FileTablespace.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/ColPartitionStoreExec.java
          • tajo-catalog/tajo-catalog-server/src/main/resources/schemas/mariadb/mariadb.xml
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Tajo-master-build #905 (See https://builds.apache.org/job/Tajo-master-build/905/ ) TAJO-1493 : Make partition pruning based on catalog informations. (blrunner: rev b5aacb917b71ae610aed9d0aa4eb76888ccc804e) tajo-plan/src/main/java/org/apache/tajo/plan/util/EvalNodeToExprConverter.java tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalog.java tajo-client/src/main/java/org/apache/tajo/client/CatalogAdminClient.java tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogService.java tajo-catalog/tajo-catalog-client/src/main/proto/CatalogProtocol.proto tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogConstants.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java tajo-core/src/main/java/org/apache/tajo/querymaster/Query.java tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto tajo-client/src/main/java/org/apache/tajo/client/TajoClientImpl.java CHANGES tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/partition/PartitionDesc.java tajo-catalog/tajo-catalog-server/src/main/resources/schemas/derby/derby.xml tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/CatalogTestingUtil.java tajo-plan/src/main/java/org/apache/tajo/plan/expr/AlgebraicUtil.java tajo-cli/src/main/java/org/apache/tajo/cli/tools/TajoDump.java tajo-common/src/main/java/org/apache/tajo/exception/UndefinedPartitionMethodException.java tajo-catalog/tajo-catalog-client/src/main/java/org/apache/tajo/catalog/AbstractCatalogClient.java tajo-core-tests/src/test/java/org/apache/tajo/engine/planner/TestEvalNodeToExprConverter.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/CatalogStore.java tajo-catalog/tajo-catalog-server/src/main/resources/schemas/oracle/oracle.xml tajo-common/src/main/java/org/apache/tajo/exception/ExceptionUtil.java tajo-catalog/tajo-catalog-server/pom.xml tajo-plan/src/main/java/org/apache/tajo/plan/util/PartitionFilterAlgebraVisitor.java tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestAlterTable.java tajo-common/src/main/proto/errors.proto tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestTablePartitions.java tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalogAgainstCaseSensitivity.java tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/DDLBuilder.java tajo-client/src/main/java/org/apache/tajo/client/CatalogAdminClientImpl.java tajo-catalog/tajo-catalog-server/src/main/resources/schemas/mysql/mysql.xml tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogStore.java tajo-catalog/tajo-catalog-drivers/tajo-hive/src/test/java/org/apache/tajo/catalog/store/TestHiveCatalogStore.java tajo-core/src/main/java/org/apache/tajo/master/TajoMasterClientService.java tajo-common/src/main/java/org/apache/tajo/exception/ReturnStateUtil.java tajo-plan/src/main/java/org/apache/tajo/plan/rewrite/rules/PartitionedTableRewriter.java tajo-common/src/main/java/org/apache/tajo/exception/ErrorMessages.java tajo-core/src/main/java/org/apache/tajo/master/exec/DDLExecutor.java tajo-catalog/tajo-catalog-server/src/main/resources/schemas/postgresql/postgresql.xml tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/FileTablespace.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/ColPartitionStoreExec.java tajo-catalog/tajo-catalog-server/src/main/resources/schemas/mariadb/mariadb.xml
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Tajo-0.11.0-build #71 (See https://builds.apache.org/job/Tajo-0.11.0-build/71/)
          TAJO-1493: Make partition pruning based on catalog informations. (blrunner: rev 51bb94d9f0d6c4598816b5bddad4f1331ccb1e58)

          • tajo-common/src/main/java/org/apache/tajo/exception/ExceptionUtil.java
          • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java
          • tajo-common/src/main/proto/errors.proto
          • tajo-catalog/tajo-catalog-server/src/main/resources/schemas/mariadb/mariadb.xml
          • tajo-core/src/main/java/org/apache/tajo/master/TajoMasterClientService.java
          • tajo-catalog/tajo-catalog-server/src/main/resources/schemas/postgresql/postgresql.xml
          • tajo-catalog/tajo-catalog-server/pom.xml
          • CHANGES
          • tajo-catalog/tajo-catalog-client/src/main/java/org/apache/tajo/catalog/AbstractCatalogClient.java
          • tajo-catalog/tajo-catalog-server/src/main/resources/schemas/mysql/mysql.xml
          • tajo-plan/src/main/java/org/apache/tajo/plan/util/PartitionFilterAlgebraVisitor.java
          • tajo-client/src/main/java/org/apache/tajo/client/CatalogAdminClient.java
          • tajo-cli/src/main/java/org/apache/tajo/cli/tools/TajoDump.java
          • tajo-catalog/tajo-catalog-server/src/main/resources/schemas/derby/derby.xml
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogService.java
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/partition/PartitionDesc.java
          • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/CatalogStore.java
          • tajo-common/src/main/java/org/apache/tajo/exception/UndefinedPartitionMethodException.java
          • tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogStore.java
          • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java
          • tajo-core/src/main/java/org/apache/tajo/querymaster/Query.java
          • tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestTablePartitions.java
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/DDLBuilder.java
          • tajo-common/src/main/java/org/apache/tajo/exception/ReturnStateUtil.java
          • tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestAlterTable.java
          • tajo-catalog/tajo-catalog-server/src/main/resources/schemas/oracle/oracle.xml
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogConstants.java
          • tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java
          • tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/FileTablespace.java
          • tajo-catalog/tajo-catalog-drivers/tajo-hive/src/test/java/org/apache/tajo/catalog/store/TestHiveCatalogStore.java
          • tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalogAgainstCaseSensitivity.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/ColPartitionStoreExec.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/rewrite/rules/PartitionedTableRewriter.java
          • tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/CatalogTestingUtil.java
          • tajo-core/src/main/java/org/apache/tajo/master/exec/DDLExecutor.java
          • tajo-core-tests/src/test/java/org/apache/tajo/engine/planner/TestEvalNodeToExprConverter.java
          • tajo-common/src/main/java/org/apache/tajo/exception/ErrorMessages.java
          • tajo-catalog/tajo-catalog-client/src/main/proto/CatalogProtocol.proto
          • tajo-plan/src/main/java/org/apache/tajo/plan/util/EvalNodeToExprConverter.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/expr/AlgebraicUtil.java
          • tajo-client/src/main/java/org/apache/tajo/client/CatalogAdminClientImpl.java
          • tajo-client/src/main/java/org/apache/tajo/client/TajoClientImpl.java
          • tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalog.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Tajo-0.11.0-build #71 (See https://builds.apache.org/job/Tajo-0.11.0-build/71/ ) TAJO-1493 : Make partition pruning based on catalog informations. (blrunner: rev 51bb94d9f0d6c4598816b5bddad4f1331ccb1e58) tajo-common/src/main/java/org/apache/tajo/exception/ExceptionUtil.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java tajo-common/src/main/proto/errors.proto tajo-catalog/tajo-catalog-server/src/main/resources/schemas/mariadb/mariadb.xml tajo-core/src/main/java/org/apache/tajo/master/TajoMasterClientService.java tajo-catalog/tajo-catalog-server/src/main/resources/schemas/postgresql/postgresql.xml tajo-catalog/tajo-catalog-server/pom.xml CHANGES tajo-catalog/tajo-catalog-client/src/main/java/org/apache/tajo/catalog/AbstractCatalogClient.java tajo-catalog/tajo-catalog-server/src/main/resources/schemas/mysql/mysql.xml tajo-plan/src/main/java/org/apache/tajo/plan/util/PartitionFilterAlgebraVisitor.java tajo-client/src/main/java/org/apache/tajo/client/CatalogAdminClient.java tajo-cli/src/main/java/org/apache/tajo/cli/tools/TajoDump.java tajo-catalog/tajo-catalog-server/src/main/resources/schemas/derby/derby.xml tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogService.java tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/partition/PartitionDesc.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/CatalogStore.java tajo-common/src/main/java/org/apache/tajo/exception/UndefinedPartitionMethodException.java tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogStore.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java tajo-core/src/main/java/org/apache/tajo/querymaster/Query.java tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestTablePartitions.java tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/DDLBuilder.java tajo-common/src/main/java/org/apache/tajo/exception/ReturnStateUtil.java tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestAlterTable.java tajo-catalog/tajo-catalog-server/src/main/resources/schemas/oracle/oracle.xml tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogConstants.java tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/FileTablespace.java tajo-catalog/tajo-catalog-drivers/tajo-hive/src/test/java/org/apache/tajo/catalog/store/TestHiveCatalogStore.java tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalogAgainstCaseSensitivity.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/ColPartitionStoreExec.java tajo-plan/src/main/java/org/apache/tajo/plan/rewrite/rules/PartitionedTableRewriter.java tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/CatalogTestingUtil.java tajo-core/src/main/java/org/apache/tajo/master/exec/DDLExecutor.java tajo-core-tests/src/test/java/org/apache/tajo/engine/planner/TestEvalNodeToExprConverter.java tajo-common/src/main/java/org/apache/tajo/exception/ErrorMessages.java tajo-catalog/tajo-catalog-client/src/main/proto/CatalogProtocol.proto tajo-plan/src/main/java/org/apache/tajo/plan/util/EvalNodeToExprConverter.java tajo-plan/src/main/java/org/apache/tajo/plan/expr/AlgebraicUtil.java tajo-client/src/main/java/org/apache/tajo/client/CatalogAdminClientImpl.java tajo-client/src/main/java/org/apache/tajo/client/TajoClientImpl.java tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalog.java

            People

            • Assignee:
              blrunner Jaehwa Jung
              Reporter:
              blrunner Jaehwa Jung
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development