Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-744 (Umbrella) ALTER TABLE ADD/DROP PARTITION statement
  3. TAJO-1346

Create dynamic partitions to CatalogStore by running insert query or CTAS query.

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.11.0
    • Component/s: Physical Operator
    • Labels:
      None

      Description

      Currently, If new partitions are added to HDFS by ColPartitionStoreExec, CatalogStore will not be aware of these partitions. So, Tajo need to load newly added partitions automatically to CatalogStore by running insert query or CTAS query.

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user blrunner opened a pull request:

          https://github.com/apache/tajo/pull/622

          TAJO-1346: Run ADD PARTITION after storing column partitioned table.

          Currently, If new partitions are added to HDFS by ColPartitionStoreExec, CatalogStore will not be aware of these partitions. So, Tajo need to run ALTER TABLE table_name ADD PARTITION on each of the newly added partitions automatically.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/blrunner/tajo TAJO-1346

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/tajo/pull/622.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #622


          commit c8634cbe225787c32ee236d217a1aa85c1254253
          Author: JaeHwa Jung <blrunner@apache.org>
          Date: 2015-07-02T13:06:07Z

          TAJO-1346: Run ADD PARTITION after storing column partitioned table.

          commit d6c93272320738bf1f524bc2cff572661130184f
          Author: JaeHwa Jung <blrunner@apache.org>
          Date: 2015-07-02T13:06:58Z

          Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1346


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user blrunner opened a pull request: https://github.com/apache/tajo/pull/622 TAJO-1346 : Run ADD PARTITION after storing column partitioned table. Currently, If new partitions are added to HDFS by ColPartitionStoreExec, CatalogStore will not be aware of these partitions. So, Tajo need to run ALTER TABLE table_name ADD PARTITION on each of the newly added partitions automatically. You can merge this pull request into a Git repository by running: $ git pull https://github.com/blrunner/tajo TAJO-1346 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/tajo/pull/622.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #622 commit c8634cbe225787c32ee236d217a1aa85c1254253 Author: JaeHwa Jung <blrunner@apache.org> Date: 2015-07-02T13:06:07Z TAJO-1346 : Run ADD PARTITION after storing column partitioned table. commit d6c93272320738bf1f524bc2cff572661130184f Author: JaeHwa Jung <blrunner@apache.org> Date: 2015-07-02T13:06:58Z Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1346
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user blrunner commented on the pull request:

          https://github.com/apache/tajo/pull/622#issuecomment-118027180

          Not yet added test cases.

          Show
          githubbot ASF GitHub Bot added a comment - Github user blrunner commented on the pull request: https://github.com/apache/tajo/pull/622#issuecomment-118027180 Not yet added test cases.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user blrunner commented on the pull request:

          https://github.com/apache/tajo/pull/622#issuecomment-119035751

          I added unit test cases. For your information, some codes would be removed after finishing TAJO-1345.

          Show
          githubbot ASF GitHub Bot added a comment - Github user blrunner commented on the pull request: https://github.com/apache/tajo/pull/622#issuecomment-119035751 I added unit test cases. For your information, some codes would be removed after finishing TAJO-1345 .
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user blrunner closed the pull request at:

          https://github.com/apache/tajo/pull/622

          Show
          githubbot ASF GitHub Bot added a comment - Github user blrunner closed the pull request at: https://github.com/apache/tajo/pull/622
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user blrunner opened a pull request:

          https://github.com/apache/tajo/pull/630

          TAJO-1346: Run ADD PARTITION after storing column partitioned table.

          Currently, If new partitions are added to HDFS by ColPartitionStoreExec, CatalogStore will not be aware of these partitions. So, Tajo need to run ALTER TABLE table_name ADD PARTITION on each of the newly added partitions automatically.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/blrunner/tajo TAJO-1346

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/tajo/pull/630.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #630


          commit 241e0668d4c2cefc57502fe0525448c00da5b442
          Author: JaeHwa Jung <blrunner@apache.org>
          Date: 2015-07-14T14:58:09Z

          TAJO-1346: Run ADD PARTITION after storing column partitioned table.


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user blrunner opened a pull request: https://github.com/apache/tajo/pull/630 TAJO-1346 : Run ADD PARTITION after storing column partitioned table. Currently, If new partitions are added to HDFS by ColPartitionStoreExec, CatalogStore will not be aware of these partitions. So, Tajo need to run ALTER TABLE table_name ADD PARTITION on each of the newly added partitions automatically. You can merge this pull request into a Git repository by running: $ git pull https://github.com/blrunner/tajo TAJO-1346 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/tajo/pull/630.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #630 commit 241e0668d4c2cefc57502fe0525448c00da5b442 Author: JaeHwa Jung <blrunner@apache.org> Date: 2015-07-14T14:58:09Z TAJO-1346 : Run ADD PARTITION after storing column partitioned table.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user blrunner commented on the pull request:

          https://github.com/apache/tajo/pull/630#issuecomment-121420637

          PartitionKey would be removed after committing TAJO-1670. And I found an issue when testing unit test cases for partitioned table. It is that table status didn't been updated when inserting into a partitioned table, if same partition directories already exist. It should be done in another issue.

          Show
          githubbot ASF GitHub Bot added a comment - Github user blrunner commented on the pull request: https://github.com/apache/tajo/pull/630#issuecomment-121420637 PartitionKey would be removed after committing TAJO-1670 . And I found an issue when testing unit test cases for partitioned table. It is that table status didn't been updated when inserting into a partitioned table, if same partition directories already exist. It should be done in another issue.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on the pull request:

          https://github.com/apache/tajo/pull/630#issuecomment-121449244

          This approach seems that each individual worker directly inserts created partitions through Catalog API. I'm concerning with too many connection problems of catalog and explosive INSERT requests simultaneously across many workers. It would be better if we can change it to a bulk insertion approach in QM. How do you think of this issue?

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on the pull request: https://github.com/apache/tajo/pull/630#issuecomment-121449244 This approach seems that each individual worker directly inserts created partitions through Catalog API. I'm concerning with too many connection problems of catalog and explosive INSERT requests simultaneously across many workers. It would be better if we can change it to a bulk insertion approach in QM. How do you think of this issue?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user blrunner commented on the pull request:

          https://github.com/apache/tajo/pull/630#issuecomment-121449543

          Thanks @hyunsik
          I agree with you and would update codes using your advice.

          Show
          githubbot ASF GitHub Bot added a comment - Github user blrunner commented on the pull request: https://github.com/apache/tajo/pull/630#issuecomment-121449543 Thanks @hyunsik I agree with you and would update codes using your advice.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user blrunner commented on the pull request:

          https://github.com/apache/tajo/pull/630#issuecomment-122177778

          Hi @hyunsik

          I've updated the patch using your comments and unit test cases finished successfully as follows.

          ```
          mvn clean install -Dtajo.catalog.store.class=org.apache.tajo.catalog.store.HiveCatalogStore
          mvn clean install -Dtajo.catalog.store.class=org.apache.tajo.catalog.store.DerbyStore
          ```

          Show
          githubbot ASF GitHub Bot added a comment - Github user blrunner commented on the pull request: https://github.com/apache/tajo/pull/630#issuecomment-122177778 Hi @hyunsik I've updated the patch using your comments and unit test cases finished successfully as follows. ``` mvn clean install -Dtajo.catalog.store.class=org.apache.tajo.catalog.store.HiveCatalogStore mvn clean install -Dtajo.catalog.store.class=org.apache.tajo.catalog.store.DerbyStore ```
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user blrunner commented on the pull request:

          https://github.com/apache/tajo/pull/630#issuecomment-124167496

          I tested to add partitions on MySQLStore as following:

          • Data Set : TPC-H 1G
          • Query Running Time
          • 10,000 partitions : 6 sec
          • 100,000 partitions: 50 sec
          • Query
            ```
            create table partitioned_lineitem3 (col1 text, col2 float8)
            partition by column(col3 int4, col4 int4) as
            select l_returnflag, l_quantity, l_orderkey, l_partkey from lineitem
            where l_orderkey < 10000;

          create table partitioned_lineitem4 (col1 text, col2 float8)
          partition by column(col3 int4, col4 int4) as
          select l_returnflag, l_quantity, l_orderkey, l_partkey from lineitem
          where l_orderkey < 100000;
          ```

          Show
          githubbot ASF GitHub Bot added a comment - Github user blrunner commented on the pull request: https://github.com/apache/tajo/pull/630#issuecomment-124167496 I tested to add partitions on MySQLStore as following: Data Set : TPC-H 1G Query Running Time 10,000 partitions : 6 sec 100,000 partitions: 50 sec Query ``` create table partitioned_lineitem3 (col1 text, col2 float8) partition by column(col3 int4, col4 int4) as select l_returnflag, l_quantity, l_orderkey, l_partkey from lineitem where l_orderkey < 10000; create table partitioned_lineitem4 (col1 text, col2 float8) partition by column(col3 int4, col4 int4) as select l_returnflag, l_quantity, l_orderkey, l_partkey from lineitem where l_orderkey < 100000; ```
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/630#discussion_r35503240

          — Diff: tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/exception/AssumedPartitionDirectoryException.java —
          @@ -18,9 +18,13 @@

          package org.apache.tajo.catalog.exception;

          -public class NoPartitionedTableException extends Exception {

          • public NoPartitionedTableException(String databaseName, String relName) { - super(String.format("ERROR: table \"%s.%s\" is not a partitioned table", databaseName, relName)); - }

            +import org.apache.tajo.error.Errors.ResultCode;
            +
            +public class AssumedPartitionDirectoryException extends CatalogException {

              • End diff –

          Its name seems to be not intuitive. Could you explain which kind of exception?

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/630#discussion_r35503240 — Diff: tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/exception/AssumedPartitionDirectoryException.java — @@ -18,9 +18,13 @@ package org.apache.tajo.catalog.exception; -public class NoPartitionedTableException extends Exception { public NoPartitionedTableException(String databaseName, String relName) { - super(String.format("ERROR: table \"%s.%s\" is not a partitioned table", databaseName, relName)); - } +import org.apache.tajo.error.Errors.ResultCode; + +public class AssumedPartitionDirectoryException extends CatalogException { End diff – Its name seems to be not intuitive. Could you explain which kind of exception?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/630#discussion_r35503343

          — Diff: tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/statistics/StatisticsUtil.java —
          @@ -83,6 +83,14 @@ public static void aggregateTableStat(TableStats result, TableStats stats) {
          }
          }

          + // If there is partitions
          — End diff –

          TableStats may be not a good place to contain added partitions because partition is not a part of statistic information. We need to improve it here or later. How do you think of it?

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/630#discussion_r35503343 — Diff: tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/statistics/StatisticsUtil.java — @@ -83,6 +83,14 @@ public static void aggregateTableStat(TableStats result, TableStats stats) { } } + // If there is partitions — End diff – TableStats may be not a good place to contain added partitions because partition is not a part of statistic information. We need to improve it here or later. How do you think of it?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/630#discussion_r35503350

          — Diff: tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/statistics/TableStats.java —
          @@ -36,6 +39,8 @@
          import java.util.List;

          public class TableStats implements ProtoObject<TableStatsProto>, Cloneable, GsonObject {
          + private static Log LOG = LogFactory.getLog(TableStats.class);
          — End diff –

          Unused logger is added.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/630#discussion_r35503350 — Diff: tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/statistics/TableStats.java — @@ -36,6 +39,8 @@ import java.util.List; public class TableStats implements ProtoObject<TableStatsProto>, Cloneable, GsonObject { + private static Log LOG = LogFactory.getLog(TableStats.class); — End diff – Unused logger is added.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/630#discussion_r35503742

          — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java —
          @@ -1245,116 +1242,118 @@ private void addNewColumn(int tableId, CatalogProtos.ColumnProto columnProto) th

          public void addPartition(int tableId, CatalogProtos.PartitionDescProto partition) throws CatalogException {
          Connection conn = null;

          • PreparedStatement pstmt = null;
          • final String ADD_PARTITION_SQL =
          • "INSERT INTO " + TB_PARTTIONS
          • + " (" + COL_TABLES_PK + ", PARTITION_NAME, PATH) VALUES (?,?,?)";
            -
          • final String ADD_PARTITION_KEYS_SQL =
          • "INSERT INTO " + TB_PARTTION_KEYS + " (" + COL_PARTITIONS_PK + ", " + COL_COLUMN_NAME + ", "
          • + COL_PARTITION_VALUE + ") VALUES (?,?,?)";
            + Statement stmt = null;

          try {
          -

          • if (LOG.isDebugEnabled()) { - LOG.debug(ADD_PARTITION_SQL); - }

            -
            conn = getConnection();

          • pstmt = conn.prepareStatement(ADD_PARTITION_SQL);
          • pstmt.setInt(1, tableId);
          • pstmt.setString(2, partition.getPartitionName());
          • pstmt.setString(3, partition.getPath());
          • pstmt.executeUpdate();
          • pstmt.close();
            + conn.setAutoCommit(false);
            + stmt = conn.createStatement();
          • if (partition.getPartitionKeysCount() > 0) {
          • pstmt = conn.prepareStatement(ADD_PARTITION_KEYS_SQL);
          • int partitionId = getPartitionId(tableId, partition.getPartitionName());
          • addPartitionKeys(pstmt, partitionId, partition);
          • pstmt.executeBatch();
            + addPartition(tableId, partition, stmt, new StringBuilder());
            + addPartitionKeys(tableId, partition, stmt, new StringBuilder());
            + stmt.executeBatch();
            +
            + if (conn != null) { + conn.commit(); }

            } catch (SQLException se) {
            + if (conn != null)

            Unknown macro: { + try { + conn.rollback(); + } catch (SQLException e) { + LOG.error(e, e); + } + }

            throw new TajoInternalError(se);
            } finally

            { - CatalogUtil.closeQuietly(pstmt); + CatalogUtil.closeQuietly(stmt); }

            }

          • public int getPartitionId(int tableId, String partitionName) throws CatalogException {
          • Connection conn = null;
          • ResultSet res = null;
          • PreparedStatement pstmt = null;
          • int retValue = -1;
            -
            + public void addPartition(int tableId, CatalogProtos.PartitionDescProto partition,
            + Statement stmt, StringBuilder sb) throws CatalogException {
            try {
          • String sql = "SELECT " + COL_PARTITIONS_PK + " FROM " + TB_PARTTIONS +
          • " WHERE " + COL_TABLES_PK + " = ? AND PARTITION_NAME = ? ";
            + sb.delete(0, sb.length());
          • if (LOG.isDebugEnabled()) { - LOG.debug(sql); - }

            -

          • conn = getConnection();
          • pstmt = conn.prepareStatement(sql);
          • pstmt.setInt(1, tableId);
          • pstmt.setString(2, partitionName);
          • res = pstmt.executeQuery();
            + sb.append("INSERT INTO ").append(TB_PARTTIONS).append(" ");
            + sb.append(" (").append(COL_TABLES_PK).append(", PARTITION_NAME, PATH)");
            + sb.append(" VALUES (").append(tableId).append(", '").append(partition.getPartitionName());
            + sb.append("' , '").append(partition.getPath()).append("')");
          • if (res.next()) { - retValue = res.getInt(1); - }

            + stmt.addBatch(sb.toString());
            } catch (SQLException se)

            { throw new TajoInternalError(se); - }

            finally

            { - CatalogUtil.closeQuietly(pstmt, res); }
          • return retValue;
            }
          • private void addPartitionKeys(PreparedStatement pstmt, int partitionId, PartitionDescProto partition) throws
          • SQLException {
          • for (int i = 0; i < partition.getPartitionKeysCount(); i++) {
          • PartitionKeyProto partitionKey = partition.getPartitionKeys;
            + public void addPartitionKeys(int tableId, CatalogProtos.PartitionDescProto partition,
              • End diff –

          StringBuilder parameter seems to be used to reuse StringBuilder object. I'd like to recommend that do not try code-level optimization until the code part actually causes bottleneck. This kind of optimization is not effective whereas it is harmful for code readability.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/630#discussion_r35503742 — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java — @@ -1245,116 +1242,118 @@ private void addNewColumn(int tableId, CatalogProtos.ColumnProto columnProto) th public void addPartition(int tableId, CatalogProtos.PartitionDescProto partition) throws CatalogException { Connection conn = null; PreparedStatement pstmt = null; final String ADD_PARTITION_SQL = "INSERT INTO " + TB_PARTTIONS + " (" + COL_TABLES_PK + ", PARTITION_NAME, PATH) VALUES (?,?,?)"; - final String ADD_PARTITION_KEYS_SQL = "INSERT INTO " + TB_PARTTION_KEYS + " (" + COL_PARTITIONS_PK + ", " + COL_COLUMN_NAME + ", " + COL_PARTITION_VALUE + ") VALUES (?,?,?)"; + Statement stmt = null; try { - if (LOG.isDebugEnabled()) { - LOG.debug(ADD_PARTITION_SQL); - } - conn = getConnection(); pstmt = conn.prepareStatement(ADD_PARTITION_SQL); pstmt.setInt(1, tableId); pstmt.setString(2, partition.getPartitionName()); pstmt.setString(3, partition.getPath()); pstmt.executeUpdate(); pstmt.close(); + conn.setAutoCommit(false); + stmt = conn.createStatement(); if (partition.getPartitionKeysCount() > 0) { pstmt = conn.prepareStatement(ADD_PARTITION_KEYS_SQL); int partitionId = getPartitionId(tableId, partition.getPartitionName()); addPartitionKeys(pstmt, partitionId, partition); pstmt.executeBatch(); + addPartition(tableId, partition, stmt, new StringBuilder()); + addPartitionKeys(tableId, partition, stmt, new StringBuilder()); + stmt.executeBatch(); + + if (conn != null) { + conn.commit(); } } catch (SQLException se) { + if (conn != null) Unknown macro: { + try { + conn.rollback(); + } catch (SQLException e) { + LOG.error(e, e); + } + } throw new TajoInternalError(se); } finally { - CatalogUtil.closeQuietly(pstmt); + CatalogUtil.closeQuietly(stmt); } } public int getPartitionId(int tableId, String partitionName) throws CatalogException { Connection conn = null; ResultSet res = null; PreparedStatement pstmt = null; int retValue = -1; - + public void addPartition(int tableId, CatalogProtos.PartitionDescProto partition, + Statement stmt, StringBuilder sb) throws CatalogException { try { String sql = "SELECT " + COL_PARTITIONS_PK + " FROM " + TB_PARTTIONS + " WHERE " + COL_TABLES_PK + " = ? AND PARTITION_NAME = ? "; + sb.delete(0, sb.length()); if (LOG.isDebugEnabled()) { - LOG.debug(sql); - } - conn = getConnection(); pstmt = conn.prepareStatement(sql); pstmt.setInt(1, tableId); pstmt.setString(2, partitionName); res = pstmt.executeQuery(); + sb.append("INSERT INTO ").append(TB_PARTTIONS).append(" "); + sb.append(" (").append(COL_TABLES_PK).append(", PARTITION_NAME, PATH)"); + sb.append(" VALUES (").append(tableId).append(", '").append(partition.getPartitionName()); + sb.append("' , '").append(partition.getPath()).append("')"); if (res.next()) { - retValue = res.getInt(1); - } + stmt.addBatch(sb.toString()); } catch (SQLException se) { throw new TajoInternalError(se); - } finally { - CatalogUtil.closeQuietly(pstmt, res); } return retValue; } private void addPartitionKeys(PreparedStatement pstmt, int partitionId, PartitionDescProto partition) throws SQLException { for (int i = 0; i < partition.getPartitionKeysCount(); i++) { PartitionKeyProto partitionKey = partition.getPartitionKeys ; + public void addPartitionKeys(int tableId, CatalogProtos.PartitionDescProto partition, End diff – StringBuilder parameter seems to be used to reuse StringBuilder object. I'd like to recommend that do not try code-level optimization until the code part actually causes bottleneck. This kind of optimization is not effective whereas it is harmful for code readability.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/630#discussion_r35504201

          — Diff: tajo-catalog/tajo-catalog-server/src/main/resources/schemas/derby/derby.xml —
          @@ -24,7 +24,7 @@

          • 2 - 2014-06-09: First versioning
          • 1- Before 2013-03-20
              • End diff –

          This histories in all schema xml files should be updated too.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/630#discussion_r35504201 — Diff: tajo-catalog/tajo-catalog-server/src/main/resources/schemas/derby/derby.xml — @@ -24,7 +24,7 @@ 2 - 2014-06-09: First versioning 1- Before 2013-03-20 End diff – This histories in all schema xml files should be updated too.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/630#discussion_r35504242

          — Diff: tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/ColPartitionStoreExec.java —
          @@ -36,9 +38,12 @@
          import org.apache.tajo.plan.logical.StoreTableNode;
          import org.apache.tajo.storage.*;
          import org.apache.tajo.unit.StorageUnit;
          +import org.apache.tajo.util.TUtil;
          — End diff –

          TUtil, URI, List are not used here.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/630#discussion_r35504242 — Diff: tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/ColPartitionStoreExec.java — @@ -36,9 +38,12 @@ import org.apache.tajo.plan.logical.StoreTableNode; import org.apache.tajo.storage.*; import org.apache.tajo.unit.StorageUnit; +import org.apache.tajo.util.TUtil; — End diff – TUtil, URI, List are not used here.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/630#discussion_r35504382

          — Diff: tajo-core/src/main/java/org/apache/tajo/querymaster/QueryMaster.java —
          @@ -32,6 +32,12 @@
          import org.apache.tajo.QueryId;

          import org.apache.tajo.TajoProtos;
          +import org.apache.tajo.catalog.CatalogService;
          +import org.apache.tajo.catalog.CatalogUtil;
          +import org.apache.tajo.catalog.TableDesc;
          +import org.apache.tajo.catalog.partition.PartitionDesc;
          — End diff –

          not used.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/630#discussion_r35504382 — Diff: tajo-core/src/main/java/org/apache/tajo/querymaster/QueryMaster.java — @@ -32,6 +32,12 @@ import org.apache.tajo.QueryId; import org.apache.tajo.TajoProtos; +import org.apache.tajo.catalog.CatalogService; +import org.apache.tajo.catalog.CatalogUtil; +import org.apache.tajo.catalog.TableDesc; +import org.apache.tajo.catalog.partition.PartitionDesc; — End diff – not used.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/630#discussion_r35504511

          — Diff: tajo-core/src/main/java/org/apache/tajo/querymaster/QueryMaster.java —
          @@ -286,6 +292,47 @@ public void stopQuery(QueryId queryId) {

          finishedQueryMasterTasksCache.put(queryId, queryMasterTask);

          + // In INSERT .. SELECT ... queries or CTAS, dynamic partitions should be added to catalog.
          + Query query = queryMasterTask.getQuery();
          + if (query.getState() == TajoProtos.QueryState.QUERY_SUCCEEDED && query.getResultDesc() != null) {
          — End diff –

          Here is not good place to do this because we cannot guarantee the query atomicity. In other words, a query is successfully completed but the partition additions can be failed. Basically, we have to guarantee both the query completion and partition addition or nothing.

          QueryCompletedTransition::finalizeQuery in Query is the best place for this.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/630#discussion_r35504511 — Diff: tajo-core/src/main/java/org/apache/tajo/querymaster/QueryMaster.java — @@ -286,6 +292,47 @@ public void stopQuery(QueryId queryId) { finishedQueryMasterTasksCache.put(queryId, queryMasterTask); + // In INSERT .. SELECT ... queries or CTAS, dynamic partitions should be added to catalog. + Query query = queryMasterTask.getQuery(); + if (query.getState() == TajoProtos.QueryState.QUERY_SUCCEEDED && query.getResultDesc() != null) { — End diff – Here is not good place to do this because we cannot guarantee the query atomicity. In other words, a query is successfully completed but the partition additions can be failed. Basically, we have to guarantee both the query completion and partition addition or nothing. QueryCompletedTransition::finalizeQuery in Query is the best place for this.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user blrunner commented on the pull request:

          https://github.com/apache/tajo/pull/630#issuecomment-125151173

          Hi @hyunsik

          Thanks for your detailed review.
          I've just updated the patch using your comments.

          Show
          githubbot ASF GitHub Bot added a comment - Github user blrunner commented on the pull request: https://github.com/apache/tajo/pull/630#issuecomment-125151173 Hi @hyunsik Thanks for your detailed review. I've just updated the patch using your comments.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user blrunner commented on the pull request:

          https://github.com/apache/tajo/pull/630#issuecomment-126214851

          I added some columns and index to ``PARTITION_KEYS`` table for getting partitions efficiently and tested on macbook successfully. The batch response time results are listed below.

          • 3,000 records: 1 ~ 2 sec
          • 30,000 records: 6 ~ 8 sec
          • 300,000 records: 60 ~ 70 sec

          Honestly, I was worried about removing foreign key of ``PARTITION_KEYS`` table because it affected to batch insert performance. But it would be good to leave it on the table because of data integrity.

          Show
          githubbot ASF GitHub Bot added a comment - Github user blrunner commented on the pull request: https://github.com/apache/tajo/pull/630#issuecomment-126214851 I added some columns and index to ``PARTITION_KEYS`` table for getting partitions efficiently and tested on macbook successfully. The batch response time results are listed below. 3,000 records: 1 ~ 2 sec 30,000 records: 6 ~ 8 sec 300,000 records: 60 ~ 70 sec Honestly, I was worried about removing foreign key of ``PARTITION_KEYS`` table because it affected to batch insert performance. But it would be good to leave it on the table because of data integrity.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/630#discussion_r35871715

          — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java —
          @@ -1065,6 +1068,47 @@ public GetTablePartitionsResponse getAllPartitions(RpcController controller, Nul
          }

          @Override
          + public ReturnState addPartitions(RpcController controller, AddPartitionsProto request)
          + throws ServiceException {
          — End diff –

          You can remove throws ServiceException.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/630#discussion_r35871715 — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java — @@ -1065,6 +1068,47 @@ public GetTablePartitionsResponse getAllPartitions(RpcController controller, Nul } @Override + public ReturnState addPartitions(RpcController controller, AddPartitionsProto request) + throws ServiceException { — End diff – You can remove throws ServiceException.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/630#discussion_r35871819

          — Diff: tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto —
          @@ -181,6 +181,7 @@ message TableStatsProto {
          optional int64 readBytes = 7;
          repeated ColumnStatsProto colStat = 8;
          optional int32 tid = 9;
          + repeated PartitionDescProto partitions = 10;
          — End diff –

          Could you keep PartitionDescProto in TableStatsProto? Other messages like TaskCompletionReport may be a better place.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/630#discussion_r35871819 — Diff: tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto — @@ -181,6 +181,7 @@ message TableStatsProto { optional int64 readBytes = 7; repeated ColumnStatsProto colStat = 8; optional int32 tid = 9; + repeated PartitionDescProto partitions = 10; — End diff – Could you keep PartitionDescProto in TableStatsProto? Other messages like TaskCompletionReport may be a better place.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/630#discussion_r35871901

          — Diff: tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogStore.java —
          @@ -957,6 +1026,54 @@ private boolean existColumn(final String databaseName ,final String tableName ,
          }

          @Override
          + public void addPartitions(String databaseName, String tableName, List<CatalogProtos.PartitionDescProto> partitions
          + , boolean ifNotExists) throws CatalogException {
          + HiveCatalogStoreClientPool.HiveCatalogStoreClient client = null;
          + List<Partition> addPartitions = TUtil.newList();
          + CatalogProtos.PartitionDescProto existingPartition = null;
          +
          + try {
          + client = clientPool.getClient();
          + for (CatalogProtos.PartitionDescProto partitionDescProto : partitions) {
          + existingPartition = getPartition(databaseName, tableName, partitionDescProto.getPartitionName());
          +
          + // Unfortunately, hive client add_partitions doesn't run as expected. The method never read the ifNotExists
          + // parameter. So, if Tajo add existing partition to Hive, it will threw AlreadyExistsException. To avoid
          — End diff –

          s/adds/add/

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/630#discussion_r35871901 — Diff: tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogStore.java — @@ -957,6 +1026,54 @@ private boolean existColumn(final String databaseName ,final String tableName , } @Override + public void addPartitions(String databaseName, String tableName, List<CatalogProtos.PartitionDescProto> partitions + , boolean ifNotExists) throws CatalogException { + HiveCatalogStoreClientPool.HiveCatalogStoreClient client = null; + List<Partition> addPartitions = TUtil.newList(); + CatalogProtos.PartitionDescProto existingPartition = null; + + try { + client = clientPool.getClient(); + for (CatalogProtos.PartitionDescProto partitionDescProto : partitions) { + existingPartition = getPartition(databaseName, tableName, partitionDescProto.getPartitionName()); + + // Unfortunately, hive client add_partitions doesn't run as expected. The method never read the ifNotExists + // parameter. So, if Tajo add existing partition to Hive, it will threw AlreadyExistsException. To avoid — End diff – s/adds/add/
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/630#discussion_r35871956

          — Diff: tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogStore.java —
          @@ -957,6 +1026,54 @@ private boolean existColumn(final String databaseName ,final String tableName ,
          }

          @Override
          + public void addPartitions(String databaseName, String tableName, List<CatalogProtos.PartitionDescProto> partitions
          + , boolean ifNotExists) throws CatalogException {
          + HiveCatalogStoreClientPool.HiveCatalogStoreClient client = null;
          + List<Partition> addPartitions = TUtil.newList();
          + CatalogProtos.PartitionDescProto existingPartition = null;
          +
          + try {
          + client = clientPool.getClient();
          + for (CatalogProtos.PartitionDescProto partitionDescProto : partitions) {
          + existingPartition = getPartition(databaseName, tableName, partitionDescProto.getPartitionName());
          +
          + // Unfortunately, hive client add_partitions doesn't run as expected. The method never read the ifNotExists
          + // parameter. So, if Tajo add existing partition to Hive, it will threw AlreadyExistsException. To avoid
          + // above error, we need filter existing partitions before call add_partitions.
          — End diff –

          ``need`` should be ``need to``.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/630#discussion_r35871956 — Diff: tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogStore.java — @@ -957,6 +1026,54 @@ private boolean existColumn(final String databaseName ,final String tableName , } @Override + public void addPartitions(String databaseName, String tableName, List<CatalogProtos.PartitionDescProto> partitions + , boolean ifNotExists) throws CatalogException { + HiveCatalogStoreClientPool.HiveCatalogStoreClient client = null; + List<Partition> addPartitions = TUtil.newList(); + CatalogProtos.PartitionDescProto existingPartition = null; + + try { + client = clientPool.getClient(); + for (CatalogProtos.PartitionDescProto partitionDescProto : partitions) { + existingPartition = getPartition(databaseName, tableName, partitionDescProto.getPartitionName()); + + // Unfortunately, hive client add_partitions doesn't run as expected. The method never read the ifNotExists + // parameter. So, if Tajo add existing partition to Hive, it will threw AlreadyExistsException. To avoid + // above error, we need filter existing partitions before call add_partitions. — End diff – ``need`` should be ``need to``.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/630#discussion_r35872100

          — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java —
          @@ -1065,6 +1068,47 @@ public GetTablePartitionsResponse getAllPartitions(RpcController controller, Nul
          }

          @Override
          + public ReturnState addPartitions(RpcController controller, AddPartitionsProto request)
          + throws ServiceException {
          +
          + TableIdentifierProto identifier = request.getTableIdentifier();
          + String databaseName = identifier.getDatabaseName();
          + String tableName = identifier.getTableName();
          +
          +
          + if (metaDictionary.isSystemDatabase(databaseName)) {
          + throw new ServiceException(databaseName + " is a system database. It does not contain any partitioned tables.");
          — End diff –

          Don't throw any exception. You can return proper ReturnState.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/630#discussion_r35872100 — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java — @@ -1065,6 +1068,47 @@ public GetTablePartitionsResponse getAllPartitions(RpcController controller, Nul } @Override + public ReturnState addPartitions(RpcController controller, AddPartitionsProto request) + throws ServiceException { + + TableIdentifierProto identifier = request.getTableIdentifier(); + String databaseName = identifier.getDatabaseName(); + String tableName = identifier.getTableName(); + + + if (metaDictionary.isSystemDatabase(databaseName)) { + throw new ServiceException(databaseName + " is a system database. It does not contain any partitioned tables."); — End diff – Don't throw any exception. You can return proper ReturnState.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/630#discussion_r35876028

          — Diff: tajo-common/src/main/java/org/apache/tajo/conf/TajoConf.java —
          @@ -290,6 +290,10 @@ public static int setDateOrder(int dateOrder) {
          PYTHON_CODE_DIR("tajo.function.python.code-dir", ""),
          PYTHON_CONTROLLER_LOG_DIR("tajo.function.python.controller.log-dir", ""),

          + // Partition
          + PARTITION_BULK_INSERT_BATCH_SIZE("tajo.partition.bulk.insert.batch.size", 1000),
          — End diff –

          I'd like to recommend four levels as follows:
          ```
          tajo.partition.dynamic.bulk-insert.batch-size
          ```

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/630#discussion_r35876028 — Diff: tajo-common/src/main/java/org/apache/tajo/conf/TajoConf.java — @@ -290,6 +290,10 @@ public static int setDateOrder(int dateOrder) { PYTHON_CODE_DIR("tajo.function.python.code-dir", ""), PYTHON_CONTROLLER_LOG_DIR("tajo.function.python.controller.log-dir", ""), + // Partition + PARTITION_BULK_INSERT_BATCH_SIZE("tajo.partition.bulk.insert.batch.size", 1000), — End diff – I'd like to recommend four levels as follows: ``` tajo.partition.dynamic.bulk-insert.batch-size ```
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/630#discussion_r35876717

          — Diff: tajo-core/src/main/java/org/apache/tajo/querymaster/QueryMaster.java —
          @@ -288,6 +292,39 @@ public void stopQuery(QueryId queryId)

          { return; }

          + // Create dynamic partitions to CatalogStore by running insert query or CTAS query.
          + Query query = queryMasterTask.getQuery();
          — End diff –

          As I mentioned in previous comment, here is not good place for it. QueryCompletedTransition in Query could be a better place.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/630#discussion_r35876717 — Diff: tajo-core/src/main/java/org/apache/tajo/querymaster/QueryMaster.java — @@ -288,6 +292,39 @@ public void stopQuery(QueryId queryId) { return; } + // Create dynamic partitions to CatalogStore by running insert query or CTAS query. + Query query = queryMasterTask.getQuery(); — End diff – As I mentioned in previous comment, here is not good place for it. QueryCompletedTransition in Query could be a better place.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/630#discussion_r35878916

          — Diff: tajo-storage/tajo-storage-common/src/main/java/org/apache/tajo/storage/TableStatistics.java —
          @@ -59,6 +61,8 @@ public TableStatistics(Schema schema)

          { comparable[i] = true; }

          }
          +
          + stat = new TableStats();
          — End diff –

          It may be a mistake.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/630#discussion_r35878916 — Diff: tajo-storage/tajo-storage-common/src/main/java/org/apache/tajo/storage/TableStatistics.java — @@ -59,6 +61,8 @@ public TableStatistics(Schema schema) { comparable[i] = true; } } + + stat = new TableStats(); — End diff – It may be a mistake.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user blrunner commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/630#discussion_r35878998

          — Diff: tajo-catalog/tajo-catalog-server/src/main/resources/schemas/derby/derby.xml —
          @@ -24,7 +24,7 @@

          • 2 - 2014-06-09: First versioning
          • 1- Before 2013-03-20
              • End diff –

          Thanks, I already added the history to all schema xml files.

          Show
          githubbot ASF GitHub Bot added a comment - Github user blrunner commented on a diff in the pull request: https://github.com/apache/tajo/pull/630#discussion_r35878998 — Diff: tajo-catalog/tajo-catalog-server/src/main/resources/schemas/derby/derby.xml — @@ -24,7 +24,7 @@ 2 - 2014-06-09: First versioning 1- Before 2013-03-20 End diff – Thanks, I already added the history to all schema xml files.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user blrunner commented on the pull request:

          https://github.com/apache/tajo/pull/630#issuecomment-126375831

          Hi @hyunsik

          Thanks for your detailed review.
          I've just updated my patch using your comments.

          Show
          githubbot ASF GitHub Bot added a comment - Github user blrunner commented on the pull request: https://github.com/apache/tajo/pull/630#issuecomment-126375831 Hi @hyunsik Thanks for your detailed review. I've just updated my patch using your comments.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on the pull request:

          https://github.com/apache/tajo/pull/630#issuecomment-126524440

          +1 The latest patch looks good to me. Thank you!

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on the pull request: https://github.com/apache/tajo/pull/630#issuecomment-126524440 +1 The latest patch looks good to me. Thank you!
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/tajo/pull/630

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/tajo/pull/630
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user blrunner commented on the pull request:

          https://github.com/apache/tajo/pull/630#issuecomment-126553824

          Thanks @hyunsik ,
          I've just committed it to the master branch.

          Show
          githubbot ASF GitHub Bot added a comment - Github user blrunner commented on the pull request: https://github.com/apache/tajo/pull/630#issuecomment-126553824 Thanks @hyunsik , I've just committed it to the master branch.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Tajo-master-CODEGEN-build #422 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/422/)
          TAJO-1346: Create dynamic partitions to CatalogStore by running insert query or CTAS query. (jaehwa) (blrunner: rev d80c32b28738a69c3a512108e65ed5c7a3e3adc6)

          • tajo-core/src/test/resources/queries/default/alter_table_drop_partition_4.sql
          • tajo-core/src/main/java/org/apache/tajo/engine/parser/SQLAnalyzer.java
          • tajo-catalog/tajo-catalog-client/src/main/java/org/apache/tajo/catalog/AbstractCatalogClient.java
          • tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogStore.java
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogService.java
          • tajo-catalog/tajo-catalog-server/src/main/resources/schemas/mysql/mysql.xml
          • tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalog.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/LogicalPlanner.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/serder/LogicalNodeDeserializer.java
          • tajo-catalog/tajo-catalog-server/src/main/resources/schemas/postgresql/postgresql.xml
          • tajo-core/src/main/java/org/apache/tajo/worker/TaskImpl.java
          • tajo-core/src/main/antlr4/org/apache/tajo/engine/parser/SQLParser.g4
          • tajo-dist/src/main/conf/catalog-site.xml.template
          • tajo-core/src/main/java/org/apache/tajo/master/exec/DDLExecutor.java
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/exception/AmbiguousPartitionDirectoryExistException.java
          • tajo-common/src/main/java/org/apache/tajo/conf/TajoConf.java
          • tajo-plan/src/main/proto/Plan.proto
          • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java
          • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/MemStore.java
          • tajo-catalog/tajo-catalog-server/src/main/resources/schemas/mariadb/mariadb.xml
          • tajo-docs/src/main/sphinx/sql_language/alter_table.rst
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/exception/NoSuchPartitionKeyException.java
          • tajo-common/src/main/proto/errors.proto
          • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/CatalogStore.java
          • tajo-core/src/main/proto/ResourceProtos.proto
          • tajo-core/src/main/java/org/apache/tajo/querymaster/TaskAttempt.java
          • tajo-core/src/test/resources/queries/default/alter_table_add_partition_5.sql
          • tajo-core/src/test/java/org/apache/tajo/engine/parser/TestSQLAnalyzer.java
          • tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto
          • tajo-catalog/tajo-catalog-server/src/main/resources/schemas/derby/derby.xml
          • tajo-core/src/test/java/org/apache/tajo/engine/query/TestAlterTable.java
          • tajo-core/src/main/java/org/apache/tajo/querymaster/Query.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/ColPartitionStoreExec.java
          • tajo-algebra/src/main/java/org/apache/tajo/algebra/AlterTable.java
          • tajo-common/src/main/java/org/apache/tajo/exception/ErrorMessages.java
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/exception/NoSuchPartitionException.java
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/exception/UndefinedPartitionMethodException.java
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/exception/AlreadyExistsAssumedPartitionDirectoryException.java
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/exception/UndefinedPartitionKeyException.java
          • tajo-core/src/test/resources/results/TestTajoCli/testAlterTableAddDropPartition.result
          • tajo-core/src/test/resources/queries/TestAlterTable/alter_table_add_partition2.sql
          • tajo-catalog/tajo-catalog-server/src/main/resources/schemas/oracle/oracle.xml
          • tajo-catalog/tajo-catalog-client/src/main/proto/CatalogProtocol.proto
          • tajo-core/src/main/java/org/apache/tajo/worker/TaskAttemptContext.java
          • tajo-core/src/test/resources/queries/TestAlterTable/alter_table_drop_partition2.sql
          • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/logical/AlterTableNode.java
          • tajo-core/src/test/java/org/apache/tajo/engine/query/TestTablePartitions.java
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/exception/NoPartitionedTableException.java
          • tajo-core/src/main/java/org/apache/tajo/querymaster/Stage.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Tajo-master-CODEGEN-build #422 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/422/ ) TAJO-1346 : Create dynamic partitions to CatalogStore by running insert query or CTAS query. (jaehwa) (blrunner: rev d80c32b28738a69c3a512108e65ed5c7a3e3adc6) tajo-core/src/test/resources/queries/default/alter_table_drop_partition_4.sql tajo-core/src/main/java/org/apache/tajo/engine/parser/SQLAnalyzer.java tajo-catalog/tajo-catalog-client/src/main/java/org/apache/tajo/catalog/AbstractCatalogClient.java tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogStore.java tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogService.java tajo-catalog/tajo-catalog-server/src/main/resources/schemas/mysql/mysql.xml tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalog.java tajo-plan/src/main/java/org/apache/tajo/plan/LogicalPlanner.java tajo-plan/src/main/java/org/apache/tajo/plan/serder/LogicalNodeDeserializer.java tajo-catalog/tajo-catalog-server/src/main/resources/schemas/postgresql/postgresql.xml tajo-core/src/main/java/org/apache/tajo/worker/TaskImpl.java tajo-core/src/main/antlr4/org/apache/tajo/engine/parser/SQLParser.g4 tajo-dist/src/main/conf/catalog-site.xml.template tajo-core/src/main/java/org/apache/tajo/master/exec/DDLExecutor.java tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/exception/AmbiguousPartitionDirectoryExistException.java tajo-common/src/main/java/org/apache/tajo/conf/TajoConf.java tajo-plan/src/main/proto/Plan.proto tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/MemStore.java tajo-catalog/tajo-catalog-server/src/main/resources/schemas/mariadb/mariadb.xml tajo-docs/src/main/sphinx/sql_language/alter_table.rst tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/exception/NoSuchPartitionKeyException.java tajo-common/src/main/proto/errors.proto tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/CatalogStore.java tajo-core/src/main/proto/ResourceProtos.proto tajo-core/src/main/java/org/apache/tajo/querymaster/TaskAttempt.java tajo-core/src/test/resources/queries/default/alter_table_add_partition_5.sql tajo-core/src/test/java/org/apache/tajo/engine/parser/TestSQLAnalyzer.java tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto tajo-catalog/tajo-catalog-server/src/main/resources/schemas/derby/derby.xml tajo-core/src/test/java/org/apache/tajo/engine/query/TestAlterTable.java tajo-core/src/main/java/org/apache/tajo/querymaster/Query.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/ColPartitionStoreExec.java tajo-algebra/src/main/java/org/apache/tajo/algebra/AlterTable.java tajo-common/src/main/java/org/apache/tajo/exception/ErrorMessages.java tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/exception/NoSuchPartitionException.java tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/exception/UndefinedPartitionMethodException.java tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/exception/AlreadyExistsAssumedPartitionDirectoryException.java tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/exception/UndefinedPartitionKeyException.java tajo-core/src/test/resources/results/TestTajoCli/testAlterTableAddDropPartition.result tajo-core/src/test/resources/queries/TestAlterTable/alter_table_add_partition2.sql tajo-catalog/tajo-catalog-server/src/main/resources/schemas/oracle/oracle.xml tajo-catalog/tajo-catalog-client/src/main/proto/CatalogProtocol.proto tajo-core/src/main/java/org/apache/tajo/worker/TaskAttemptContext.java tajo-core/src/test/resources/queries/TestAlterTable/alter_table_drop_partition2.sql tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java tajo-plan/src/main/java/org/apache/tajo/plan/logical/AlterTableNode.java tajo-core/src/test/java/org/apache/tajo/engine/query/TestTablePartitions.java tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/exception/NoPartitionedTableException.java tajo-core/src/main/java/org/apache/tajo/querymaster/Stage.java
          Hide
          hudson Hudson added a comment -

          ABORTED: Integrated in Tajo-master-build #784 (See https://builds.apache.org/job/Tajo-master-build/784/)
          TAJO-1346: Create dynamic partitions to CatalogStore by running insert query or CTAS query. (jaehwa) (blrunner: rev d80c32b28738a69c3a512108e65ed5c7a3e3adc6)

          • tajo-catalog/tajo-catalog-client/src/main/java/org/apache/tajo/catalog/AbstractCatalogClient.java
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/exception/UndefinedPartitionKeyException.java
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/exception/NoSuchPartitionException.java
          • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java
          • tajo-core/src/test/resources/queries/default/alter_table_drop_partition_4.sql
          • tajo-core/src/main/antlr4/org/apache/tajo/engine/parser/SQLParser.g4
          • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/CatalogStore.java
          • tajo-docs/src/main/sphinx/sql_language/alter_table.rst
          • tajo-plan/src/main/java/org/apache/tajo/plan/logical/AlterTableNode.java
          • tajo-common/src/main/java/org/apache/tajo/conf/TajoConf.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/ColPartitionStoreExec.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/LogicalPlanner.java
          • tajo-core/src/main/proto/ResourceProtos.proto
          • tajo-core/src/main/java/org/apache/tajo/worker/TaskImpl.java
          • tajo-plan/src/main/proto/Plan.proto
          • tajo-core/src/test/resources/queries/TestAlterTable/alter_table_add_partition2.sql
          • tajo-catalog/tajo-catalog-server/src/main/resources/schemas/postgresql/postgresql.xml
          • tajo-catalog/tajo-catalog-server/src/main/resources/schemas/mariadb/mariadb.xml
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/exception/NoSuchPartitionKeyException.java
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/exception/AlreadyExistsAssumedPartitionDirectoryException.java
          • tajo-catalog/tajo-catalog-server/src/main/resources/schemas/derby/derby.xml
          • tajo-common/src/main/proto/errors.proto
          • tajo-core/src/test/resources/queries/default/alter_table_add_partition_5.sql
          • tajo-core/src/test/java/org/apache/tajo/engine/query/TestTablePartitions.java
          • tajo-core/src/test/resources/queries/TestAlterTable/alter_table_drop_partition2.sql
          • tajo-algebra/src/main/java/org/apache/tajo/algebra/AlterTable.java
          • tajo-dist/src/main/conf/catalog-site.xml.template
          • tajo-core/src/main/java/org/apache/tajo/worker/TaskAttemptContext.java
          • tajo-catalog/tajo-catalog-client/src/main/proto/CatalogProtocol.proto
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/exception/AmbiguousPartitionDirectoryExistException.java
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogService.java
          • tajo-core/src/main/java/org/apache/tajo/engine/parser/SQLAnalyzer.java
          • tajo-core/src/test/java/org/apache/tajo/engine/parser/TestSQLAnalyzer.java
          • tajo-core/src/test/java/org/apache/tajo/engine/query/TestAlterTable.java
          • tajo-catalog/tajo-catalog-server/src/main/resources/schemas/mysql/mysql.xml
          • tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogStore.java
          • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/MemStore.java
          • tajo-core/src/main/java/org/apache/tajo/querymaster/Query.java
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/exception/UndefinedPartitionMethodException.java
          • tajo-common/src/main/java/org/apache/tajo/exception/ErrorMessages.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/serder/LogicalNodeDeserializer.java
          • tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto
          • tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalog.java
          • tajo-core/src/main/java/org/apache/tajo/querymaster/TaskAttempt.java
          • tajo-core/src/test/resources/results/TestTajoCli/testAlterTableAddDropPartition.result
          • tajo-core/src/main/java/org/apache/tajo/querymaster/Stage.java
          • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java
          • tajo-core/src/main/java/org/apache/tajo/master/exec/DDLExecutor.java
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/exception/NoPartitionedTableException.java
          • tajo-catalog/tajo-catalog-server/src/main/resources/schemas/oracle/oracle.xml
          Show
          hudson Hudson added a comment - ABORTED: Integrated in Tajo-master-build #784 (See https://builds.apache.org/job/Tajo-master-build/784/ ) TAJO-1346 : Create dynamic partitions to CatalogStore by running insert query or CTAS query. (jaehwa) (blrunner: rev d80c32b28738a69c3a512108e65ed5c7a3e3adc6) tajo-catalog/tajo-catalog-client/src/main/java/org/apache/tajo/catalog/AbstractCatalogClient.java tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/exception/UndefinedPartitionKeyException.java tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/exception/NoSuchPartitionException.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java tajo-core/src/test/resources/queries/default/alter_table_drop_partition_4.sql tajo-core/src/main/antlr4/org/apache/tajo/engine/parser/SQLParser.g4 tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/CatalogStore.java tajo-docs/src/main/sphinx/sql_language/alter_table.rst tajo-plan/src/main/java/org/apache/tajo/plan/logical/AlterTableNode.java tajo-common/src/main/java/org/apache/tajo/conf/TajoConf.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/ColPartitionStoreExec.java tajo-plan/src/main/java/org/apache/tajo/plan/LogicalPlanner.java tajo-core/src/main/proto/ResourceProtos.proto tajo-core/src/main/java/org/apache/tajo/worker/TaskImpl.java tajo-plan/src/main/proto/Plan.proto tajo-core/src/test/resources/queries/TestAlterTable/alter_table_add_partition2.sql tajo-catalog/tajo-catalog-server/src/main/resources/schemas/postgresql/postgresql.xml tajo-catalog/tajo-catalog-server/src/main/resources/schemas/mariadb/mariadb.xml tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/exception/NoSuchPartitionKeyException.java tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/exception/AlreadyExistsAssumedPartitionDirectoryException.java tajo-catalog/tajo-catalog-server/src/main/resources/schemas/derby/derby.xml tajo-common/src/main/proto/errors.proto tajo-core/src/test/resources/queries/default/alter_table_add_partition_5.sql tajo-core/src/test/java/org/apache/tajo/engine/query/TestTablePartitions.java tajo-core/src/test/resources/queries/TestAlterTable/alter_table_drop_partition2.sql tajo-algebra/src/main/java/org/apache/tajo/algebra/AlterTable.java tajo-dist/src/main/conf/catalog-site.xml.template tajo-core/src/main/java/org/apache/tajo/worker/TaskAttemptContext.java tajo-catalog/tajo-catalog-client/src/main/proto/CatalogProtocol.proto tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/exception/AmbiguousPartitionDirectoryExistException.java tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogService.java tajo-core/src/main/java/org/apache/tajo/engine/parser/SQLAnalyzer.java tajo-core/src/test/java/org/apache/tajo/engine/parser/TestSQLAnalyzer.java tajo-core/src/test/java/org/apache/tajo/engine/query/TestAlterTable.java tajo-catalog/tajo-catalog-server/src/main/resources/schemas/mysql/mysql.xml tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogStore.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/MemStore.java tajo-core/src/main/java/org/apache/tajo/querymaster/Query.java tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/exception/UndefinedPartitionMethodException.java tajo-common/src/main/java/org/apache/tajo/exception/ErrorMessages.java tajo-plan/src/main/java/org/apache/tajo/plan/serder/LogicalNodeDeserializer.java tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalog.java tajo-core/src/main/java/org/apache/tajo/querymaster/TaskAttempt.java tajo-core/src/test/resources/results/TestTajoCli/testAlterTableAddDropPartition.result tajo-core/src/main/java/org/apache/tajo/querymaster/Stage.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java tajo-core/src/main/java/org/apache/tajo/master/exec/DDLExecutor.java tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/exception/NoPartitionedTableException.java tajo-catalog/tajo-catalog-server/src/main/resources/schemas/oracle/oracle.xml

            People

            • Assignee:
              blrunner Jaehwa Jung
              Reporter:
              blrunner Jaehwa Jung
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development