Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-1213

Implement CatalogStore::updateTableStats

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.10.0
    • Component/s: Catalog
    • Labels:
      None

      Description

      Currently, there is no method to update existing table information at CatalogStore. So, if user execute an insert query, tajo drop a table and then tajo create a table. It doesn't matter to tajo own CatalogStore like DerbyStore, MySQLStore. But it is a critical issue to HCatalogStore. if tajo drop a existing partitioned table, users will lose existing partition informations. As a result, they should repair table partition against their will. As you know, hive msck command can affect hive metastore performance.

        Activity

        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user blrunner commented on the pull request:

        https://github.com/apache/tajo/pull/283#issuecomment-65736023

        I'll create new pull request.

        Show
        githubbot ASF GitHub Bot added a comment - Github user blrunner commented on the pull request: https://github.com/apache/tajo/pull/283#issuecomment-65736023 I'll create new pull request.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user blrunner closed the pull request at:

        https://github.com/apache/tajo/pull/283

        Show
        githubbot ASF GitHub Bot added a comment - Github user blrunner closed the pull request at: https://github.com/apache/tajo/pull/283
        Hide
        githubbot ASF GitHub Bot added a comment -

        GitHub user blrunner opened a pull request:

        https://github.com/apache/tajo/pull/285

        TAJO-1213: Implement CatalogStore::updateTableStats

        I added a method to update TableStats at CatalogStore. For reference, all unit cases for physical operators verify output row numbers and it finished successfully with my patch. So, I didn't implement unit test cases. In addition, HCatalogStore::updateTableStats is dummy method. Because hive don't provide no api to update table row numbers.

        You can merge this pull request into a Git repository by running:

        $ git pull https://github.com/blrunner/tajo TAJO-1213

        Alternatively you can review and apply these changes as the patch at:

        https://github.com/apache/tajo/pull/285.patch

        To close this pull request, make a commit to your master/trunk branch
        with (at least) the following in the commit message:

        This closes #285


        commit 400130ef27056c08d93a6385c7c7a6d74c092e5f
        Author: JaeHwa Jung <blrunner@apache.org>
        Date: 2014-12-04T09:08:48Z

        TAJO-1213: Implement CatalogStore::updateTableStats.

        commit b68195c803bbf09556e78e6194a23f8173fcaa32
        Author: JaeHwa Jung <blrunner@apache.org>
        Date: 2014-12-04T09:15:12Z

        Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1213


        Show
        githubbot ASF GitHub Bot added a comment - GitHub user blrunner opened a pull request: https://github.com/apache/tajo/pull/285 TAJO-1213 : Implement CatalogStore::updateTableStats I added a method to update TableStats at CatalogStore. For reference, all unit cases for physical operators verify output row numbers and it finished successfully with my patch. So, I didn't implement unit test cases. In addition, HCatalogStore::updateTableStats is dummy method. Because hive don't provide no api to update table row numbers. You can merge this pull request into a Git repository by running: $ git pull https://github.com/blrunner/tajo TAJO-1213 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/tajo/pull/285.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #285 commit 400130ef27056c08d93a6385c7c7a6d74c092e5f Author: JaeHwa Jung <blrunner@apache.org> Date: 2014-12-04T09:08:48Z TAJO-1213 : Implement CatalogStore::updateTableStats. commit b68195c803bbf09556e78e6194a23f8173fcaa32 Author: JaeHwa Jung <blrunner@apache.org> Date: 2014-12-04T09:15:12Z Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1213
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user jihoonson commented on a diff in the pull request:

        https://github.com/apache/tajo/pull/285#discussion_r21351579

        — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java —
        @@ -823,6 +823,74 @@ public void createTable(final CatalogProtos.TableDescProto table) throws Catalog
        }

        @Override
        + public void updateTableStats(final CatalogProtos.UpdateTableStatsProto statsProto) throws
        + CatalogException {
        + Connection conn = null;
        + PreparedStatement pstmt = null;
        + ResultSet res = null;
        +
        + try {
        + conn = getConnection();
        + conn.setAutoCommit(false);
        +
        + String[] splitted = CatalogUtil.splitTableName(statsProto.getTableName());
        + if (splitted.length == 1) {
        + throw new IllegalArgumentException("createTable() requires a qualified table name, but it is \""
        — End diff –

        The exception message should be improved to print a proper function name.

        Show
        githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/285#discussion_r21351579 — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java — @@ -823,6 +823,74 @@ public void createTable(final CatalogProtos.TableDescProto table) throws Catalog } @Override + public void updateTableStats(final CatalogProtos.UpdateTableStatsProto statsProto) throws + CatalogException { + Connection conn = null; + PreparedStatement pstmt = null; + ResultSet res = null; + + try { + conn = getConnection(); + conn.setAutoCommit(false); + + String[] splitted = CatalogUtil.splitTableName(statsProto.getTableName()); + if (splitted.length == 1) { + throw new IllegalArgumentException("createTable() requires a qualified table name, but it is \"" — End diff – The exception message should be improved to print a proper function name.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user jihoonson commented on the pull request:

        https://github.com/apache/tajo/pull/285#issuecomment-65738424

        +1, this patch looks good to me.
        I've left only one trivial comment.
        BTW, I think this issue is quite related to https://issues.apache.org/jira/browse/TAJO-1091.
        Actually, I have a plan to start this kind of work, but you already did.
        So, thanks for your work!!

        Show
        githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on the pull request: https://github.com/apache/tajo/pull/285#issuecomment-65738424 +1, this patch looks good to me. I've left only one trivial comment. BTW, I think this issue is quite related to https://issues.apache.org/jira/browse/TAJO-1091 . Actually, I have a plan to start this kind of work, but you already did. So, thanks for your work!!
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user asfgit closed the pull request at:

        https://github.com/apache/tajo/pull/285

        Show
        githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/tajo/pull/285
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user blrunner commented on the pull request:

        https://github.com/apache/tajo/pull/285#issuecomment-65928618

        Thanks @jihoonson.
        I've just committed it to master branch.
        And we need to improve CatalogStore API continuously.

        Show
        githubbot ASF GitHub Bot added a comment - Github user blrunner commented on the pull request: https://github.com/apache/tajo/pull/285#issuecomment-65928618 Thanks @jihoonson. I've just committed it to master branch. And we need to improve CatalogStore API continuously.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Tajo-master-build #484 (See https://builds.apache.org/job/Tajo-master-build/484/)
        TAJO-1213: Implement CatalogStore::updateTableStats. (jaehwa) (blrunner: rev 95cf4b9432a02fdbf9880b204c3db718e2bd2468)

        • tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto
        • tajo-catalog/tajo-catalog-client/src/main/java/org/apache/tajo/catalog/AbstractCatalogClient.java
        • CHANGES
        • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java
        • tajo-catalog/tajo-catalog-client/src/main/proto/CatalogProtocol.proto
        • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java
        • tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogStore.java
        • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/CatalogStore.java
        • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogService.java
        • tajo-core/src/main/java/org/apache/tajo/master/querymaster/Query.java
        • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/MemStore.java
        • tajo-core/src/main/java/org/apache/tajo/master/GlobalEngine.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Tajo-master-build #484 (See https://builds.apache.org/job/Tajo-master-build/484/ ) TAJO-1213 : Implement CatalogStore::updateTableStats. (jaehwa) (blrunner: rev 95cf4b9432a02fdbf9880b204c3db718e2bd2468) tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto tajo-catalog/tajo-catalog-client/src/main/java/org/apache/tajo/catalog/AbstractCatalogClient.java CHANGES tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java tajo-catalog/tajo-catalog-client/src/main/proto/CatalogProtocol.proto tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogStore.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/CatalogStore.java tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogService.java tajo-core/src/main/java/org/apache/tajo/master/querymaster/Query.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/MemStore.java tajo-core/src/main/java/org/apache/tajo/master/GlobalEngine.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Tajo-master-CODEGEN-build #125 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/125/)
        TAJO-1213: Implement CatalogStore::updateTableStats. (jaehwa) (blrunner: rev 95cf4b9432a02fdbf9880b204c3db718e2bd2468)

        • tajo-core/src/main/java/org/apache/tajo/master/GlobalEngine.java
        • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogService.java
        • tajo-catalog/tajo-catalog-client/src/main/proto/CatalogProtocol.proto
        • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java
        • CHANGES
        • tajo-core/src/main/java/org/apache/tajo/master/querymaster/Query.java
        • tajo-catalog/tajo-catalog-client/src/main/java/org/apache/tajo/catalog/AbstractCatalogClient.java
        • tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogStore.java
        • tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto
        • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/CatalogStore.java
        • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java
        • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/MemStore.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Tajo-master-CODEGEN-build #125 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/125/ ) TAJO-1213 : Implement CatalogStore::updateTableStats. (jaehwa) (blrunner: rev 95cf4b9432a02fdbf9880b204c3db718e2bd2468) tajo-core/src/main/java/org/apache/tajo/master/GlobalEngine.java tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogService.java tajo-catalog/tajo-catalog-client/src/main/proto/CatalogProtocol.proto tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java CHANGES tajo-core/src/main/java/org/apache/tajo/master/querymaster/Query.java tajo-catalog/tajo-catalog-client/src/main/java/org/apache/tajo/catalog/AbstractCatalogClient.java tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogStore.java tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/CatalogStore.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/MemStore.java

          People

          • Assignee:
            blrunner Jaehwa Jung
            Reporter:
            blrunner Jaehwa Jung
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development