Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-1674

Validation of CTAS schema mismatch

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.11.0
    • Component/s: Query Validation
    • Labels:
      None

      Description

      See the title.
      You can reproduce this error as follows.

      default> create table partitioned_nation (n_name text, n_comment text) partition by column (n_nationkey int8, n_regionkey int8) as select * from nation;
      
      default> \d partitioned_nation
      
      table name: default.partitioned_nation
      table uri: hdfs://localhost:7020/tajo/warehouse/default/partitioned_nation
      store type: CSV
      number of rows: 25
      volume: 267 B
      Options: 
          'text.delimiter'='|'
      
      schema: 
      n_name  TEXT
      n_comment   TEXT
      
      Partitions: 
      type:COLUMN
      columns::n_nationkey (INT8), n_regionkey (INT8)
      
      default> \dfs -ls hdfs://localhost:7020/tajo/warehouse/default/partitioned_nation/n_nationkey=0
      Found 5 items
      drwxr-xr-x   - jihoonson supergroup          0 2015-07-06 16:29 hdfs://localhost:7020/tajo/warehouse/default/partitioned_nation/n_nationkey=0/n_regionkey= haggle. carefully final deposits detect slyly agai
      drwxr-xr-x   - jihoonson supergroup          0 2015-07-06 16:29 hdfs://localhost:7020/tajo/warehouse/default/partitioned_nation/n_nationkey=0/n_regionkey= pending excuses haggle furiously deposits. pending, express pinto beans wake fluffily past t
      drwxr-xr-x   - jihoonson supergroup          0 2015-07-06 16:29 hdfs://localhost:7020/tajo/warehouse/default/partitioned_nation/n_nationkey=0/n_regionkey=rns. blithely bold courts among the closely regular packages use furiously bold platelets%3F
      drwxr-xr-x   - jihoonson supergroup          0 2015-07-06 16:29 hdfs://localhost:7020/tajo/warehouse/default/partitioned_nation/n_nationkey=0/n_regionkey=s. ironic, unusual asymptotes wake blithely r
      drwxr-xr-x   - jihoonson supergroup          0 2015-07-06 16:29 hdfs://localhost:7020/tajo/warehouse/default/partitioned_nation/n_nationkey=0/n_regionkey=ven packages wake quickly. regu
      

        Activity

        Hide
        githubbot ASF GitHub Bot added a comment -

        GitHub user hyunsik opened a pull request:

        https://github.com/apache/tajo/pull/726

        TAJO-1674: Validation of CTAS schema mismatch.

        You can merge this pull request into a Git repository by running:

        $ git pull https://github.com/hyunsik/tajo TAJO-1674

        Alternatively you can review and apply these changes as the patch at:

        https://github.com/apache/tajo/pull/726.patch

        To close this pull request, make a commit to your master/trunk branch
        with (at least) the following in the commit message:

        This closes #726


        commit 2c576a5e4a8f4a232bd429441476e317edf6a4f8
        Author: Hyunsik Choi <hyunsik@apache.org>
        Date: 2015-09-04T03:32:44Z

        TAJO-1674: Validation of CTAS schema mismatch.


        Show
        githubbot ASF GitHub Bot added a comment - GitHub user hyunsik opened a pull request: https://github.com/apache/tajo/pull/726 TAJO-1674 : Validation of CTAS schema mismatch. You can merge this pull request into a Git repository by running: $ git pull https://github.com/hyunsik/tajo TAJO-1674 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/tajo/pull/726.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #726 commit 2c576a5e4a8f4a232bd429441476e317edf6a4f8 Author: Hyunsik Choi <hyunsik@apache.org> Date: 2015-09-04T03:32:44Z TAJO-1674 : Validation of CTAS schema mismatch.
        Hide
        hyunsik Hyunsik Choi added a comment -

        This is a kind of query validation error. I added the validation rule.

        Show
        hyunsik Hyunsik Choi added a comment - This is a kind of query validation error. I added the validation rule.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user hyunsik commented on the pull request:

        https://github.com/apache/tajo/pull/726#issuecomment-137654265

        Fixed test failures and rebased.

        Show
        githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on the pull request: https://github.com/apache/tajo/pull/726#issuecomment-137654265 Fixed test failures and rebased.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user blrunner commented on a diff in the pull request:

        https://github.com/apache/tajo/pull/726#discussion_r38835276

        — Diff: tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java —
        @@ -929,9 +933,16 @@ public static AlterTableDesc addOrDropPartition(String tableName, String[] colum
        TUtil.putToNestedMap(OPERATION_CASTING_MAP, Type.FLOAT8, Type.TEXT, Type.TEXT);

        TUtil.putToNestedMap(OPERATION_CASTING_MAP, Type.TEXT, Type.TIMESTAMP, Type.TIMESTAMP);
        + TUtil.putToNestedMap(OPERATION_CASTING_MAP, Type.TEXT, Type.TEXT, Type.TEXT);
        — End diff –

        We might need time type and date type.

        Show
        githubbot ASF GitHub Bot added a comment - Github user blrunner commented on a diff in the pull request: https://github.com/apache/tajo/pull/726#discussion_r38835276 — Diff: tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java — @@ -929,9 +933,16 @@ public static AlterTableDesc addOrDropPartition(String tableName, String[] colum TUtil.putToNestedMap(OPERATION_CASTING_MAP, Type.FLOAT8, Type.TEXT, Type.TEXT); TUtil.putToNestedMap(OPERATION_CASTING_MAP, Type.TEXT, Type.TIMESTAMP, Type.TIMESTAMP); + TUtil.putToNestedMap(OPERATION_CASTING_MAP, Type.TEXT, Type.TEXT, Type.TEXT); — End diff – We might need time type and date type.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user blrunner commented on a diff in the pull request:

        https://github.com/apache/tajo/pull/726#discussion_r38835474

        — Diff: tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestTablePartitions.java —
        @@ -211,7 +208,7 @@ public final void testCreateColumnPartitionedTableWithSelectedColumns() throws E

        TableDesc tableDesc = catalog.getTableDesc(DEFAULT_DATABASE_NAME, tableName);
        verifyPartitionDirectoryFromCatalog(DEFAULT_DATABASE_NAME, tableName, new String[]

        {"key"}

        ,

        • tableDesc.getStats().getNumRows());
          + tableDesc.getStats().getNumRows());
            • End diff –

        This class seems to have unnecessary modifications.

        Show
        githubbot ASF GitHub Bot added a comment - Github user blrunner commented on a diff in the pull request: https://github.com/apache/tajo/pull/726#discussion_r38835474 — Diff: tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestTablePartitions.java — @@ -211,7 +208,7 @@ public final void testCreateColumnPartitionedTableWithSelectedColumns() throws E TableDesc tableDesc = catalog.getTableDesc(DEFAULT_DATABASE_NAME, tableName); verifyPartitionDirectoryFromCatalog(DEFAULT_DATABASE_NAME, tableName, new String[] {"key"} , tableDesc.getStats().getNumRows()); + tableDesc.getStats().getNumRows()); End diff – This class seems to have unnecessary modifications.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user hyunsik commented on the pull request:

        https://github.com/apache/tajo/pull/726#issuecomment-138293860

        I reflected the comments and rebased.

        Show
        githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on the pull request: https://github.com/apache/tajo/pull/726#issuecomment-138293860 I reflected the comments and rebased.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user jihoonson commented on the pull request:

        https://github.com/apache/tajo/pull/726#issuecomment-138447880

        +1 ship it!

        Show
        githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on the pull request: https://github.com/apache/tajo/pull/726#issuecomment-138447880 +1 ship it!
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user asfgit closed the pull request at:

        https://github.com/apache/tajo/pull/726

        Show
        githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/tajo/pull/726
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Tajo-master-CODEGEN-build #490 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/490/)
        TAJO-1674: Validation of CTAS schema mismatch. (hyunsik: rev bdd8e76855eb0a79f22c67832edc5ce535a1a938)

        • CHANGES
        • tajo-core-tests/src/test/resources/queries/TestCreateTable/negative/type_mismatch.sql
        • tajo-plan/src/main/java/org/apache/tajo/plan/verifier/LogicalPlanVerifier.java
        • tajo-plan/src/main/java/org/apache/tajo/plan/verifier/SyntaxErrorUtil.java
        • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java
        • tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestTablePartitions.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Tajo-master-CODEGEN-build #490 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/490/ ) TAJO-1674 : Validation of CTAS schema mismatch. (hyunsik: rev bdd8e76855eb0a79f22c67832edc5ce535a1a938) CHANGES tajo-core-tests/src/test/resources/queries/TestCreateTable/negative/type_mismatch.sql tajo-plan/src/main/java/org/apache/tajo/plan/verifier/LogicalPlanVerifier.java tajo-plan/src/main/java/org/apache/tajo/plan/verifier/SyntaxErrorUtil.java tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestTablePartitions.java
        Hide
        hyunsik Hyunsik Choi added a comment -

        I just committed it to master and branch-0.11.0 branches.

        Show
        hyunsik Hyunsik Choi added a comment - I just committed it to master and branch-0.11.0 branches.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Tajo-master-build #848 (See https://builds.apache.org/job/Tajo-master-build/848/)
        TAJO-1674: Validation of CTAS schema mismatch. (hyunsik: rev bdd8e76855eb0a79f22c67832edc5ce535a1a938)

        • tajo-plan/src/main/java/org/apache/tajo/plan/verifier/SyntaxErrorUtil.java
        • tajo-core-tests/src/test/resources/queries/TestCreateTable/negative/type_mismatch.sql
        • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java
        • tajo-plan/src/main/java/org/apache/tajo/plan/verifier/LogicalPlanVerifier.java
        • CHANGES
        • tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestTablePartitions.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Tajo-master-build #848 (See https://builds.apache.org/job/Tajo-master-build/848/ ) TAJO-1674 : Validation of CTAS schema mismatch. (hyunsik: rev bdd8e76855eb0a79f22c67832edc5ce535a1a938) tajo-plan/src/main/java/org/apache/tajo/plan/verifier/SyntaxErrorUtil.java tajo-core-tests/src/test/resources/queries/TestCreateTable/negative/type_mismatch.sql tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java tajo-plan/src/main/java/org/apache/tajo/plan/verifier/LogicalPlanVerifier.java CHANGES tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestTablePartitions.java
        Hide
        hudson Hudson added a comment -

        ABORTED: Integrated in Tajo-0.11.0-build #25 (See https://builds.apache.org/job/Tajo-0.11.0-build/25/)
        TAJO-1674: Validation of CTAS schema mismatch. (hyunsik: rev 7b27c8ce9768aa56ed584757caa33f44d56a14ab)

        • CHANGES
        • tajo-plan/src/main/java/org/apache/tajo/plan/verifier/LogicalPlanVerifier.java
        • tajo-core-tests/src/test/resources/queries/TestCreateTable/negative/type_mismatch.sql
        • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java
        • tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestTablePartitions.java
        • tajo-plan/src/main/java/org/apache/tajo/plan/verifier/SyntaxErrorUtil.java
        Show
        hudson Hudson added a comment - ABORTED: Integrated in Tajo-0.11.0-build #25 (See https://builds.apache.org/job/Tajo-0.11.0-build/25/ ) TAJO-1674 : Validation of CTAS schema mismatch. (hyunsik: rev 7b27c8ce9768aa56ed584757caa33f44d56a14ab) CHANGES tajo-plan/src/main/java/org/apache/tajo/plan/verifier/LogicalPlanVerifier.java tajo-core-tests/src/test/resources/queries/TestCreateTable/negative/type_mismatch.sql tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestTablePartitions.java tajo-plan/src/main/java/org/apache/tajo/plan/verifier/SyntaxErrorUtil.java

          People

          • Assignee:
            hyunsik Hyunsik Choi
            Reporter:
            jihoonson Jihoon Son
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development