[TAJO-1674] Validation of CTAS schema mismatch - ASF JIRA

Details

Type: Bug
Status: Resolved
Priority: Blocker
Resolution: Fixed
Affects Version/s: None
Fix Version/s: 0.11.0
Component/s: Query Validation
Labels:
None

Description

See the title.
You can reproduce this error as follows.

default> create table partitioned_nation (n_name text, n_comment text) partition by column (n_nationkey int8, n_regionkey int8) as select * from nation;

default> \d partitioned_nation

table name: default.partitioned_nation
table uri: hdfs://localhost:7020/tajo/warehouse/default/partitioned_nation
store type: CSV
number of rows: 25
volume: 267 B
Options: 
    'text.delimiter'='|'

schema: 
n_name  TEXT
n_comment   TEXT

Partitions: 
type:COLUMN
columns::n_nationkey (INT8), n_regionkey (INT8)

default> \dfs -ls hdfs://localhost:7020/tajo/warehouse/default/partitioned_nation/n_nationkey=0
Found 5 items
drwxr-xr-x   - jihoonson supergroup          0 2015-07-06 16:29 hdfs://localhost:7020/tajo/warehouse/default/partitioned_nation/n_nationkey=0/n_regionkey= haggle. carefully final deposits detect slyly agai
drwxr-xr-x   - jihoonson supergroup          0 2015-07-06 16:29 hdfs://localhost:7020/tajo/warehouse/default/partitioned_nation/n_nationkey=0/n_regionkey= pending excuses haggle furiously deposits. pending, express pinto beans wake fluffily past t
drwxr-xr-x   - jihoonson supergroup          0 2015-07-06 16:29 hdfs://localhost:7020/tajo/warehouse/default/partitioned_nation/n_nationkey=0/n_regionkey=rns. blithely bold courts among the closely regular packages use furiously bold platelets%3F
drwxr-xr-x   - jihoonson supergroup          0 2015-07-06 16:29 hdfs://localhost:7020/tajo/warehouse/default/partitioned_nation/n_nationkey=0/n_regionkey=s. ironic, unusual asymptotes wake blithely r
drwxr-xr-x   - jihoonson supergroup          0 2015-07-06 16:29 hdfs://localhost:7020/tajo/warehouse/default/partitioned_nation/n_nationkey=0/n_regionkey=ven packages wake quickly. regu

Activity

Ascending order - Click to sort in descending order

Hide

Permalink

ASF GitHub Bot added a comment - 04/Sep/15 03:34

GitHub user hyunsik opened a pull request:

https://github.com/apache/tajo/pull/726

~~TAJO-1674~~: Validation of CTAS schema mismatch.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hyunsik/tajo ~~TAJO-1674~~

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/tajo/pull/726.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #726

commit 2c576a5e4a8f4a232bd429441476e317edf6a4f8
Author: Hyunsik Choi <hyunsik@apache.org>
Date: 2015-09-04T03:32:44Z

~~TAJO-1674~~: Validation of CTAS schema mismatch.

Show

ASF GitHub Bot added a comment - 04/Sep/15 03:34 GitHub user hyunsik opened a pull request: https://github.com/apache/tajo/pull/726 TAJO-1674 : Validation of CTAS schema mismatch. You can merge this pull request into a Git repository by running: $ git pull https://github.com/hyunsik/tajo TAJO-1674 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/tajo/pull/726.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #726 commit 2c576a5e4a8f4a232bd429441476e317edf6a4f8 Author: Hyunsik Choi <hyunsik@apache.org> Date: 2015-09-04T03:32:44Z TAJO-1674 : Validation of CTAS schema mismatch.

Hide

Permalink

Hyunsik Choi added a comment - 04/Sep/15 03:35

This is a kind of query validation error. I added the validation rule.

Show

Hyunsik Choi added a comment - 04/Sep/15 03:35 This is a kind of query validation error. I added the validation rule.

Hide

Permalink

ASF GitHub Bot added a comment - 04/Sep/15 05:48

Github user hyunsik commented on the pull request:

https://github.com/apache/tajo/pull/726#issuecomment-137654265

Fixed test failures and rebased.

Show

ASF GitHub Bot added a comment - 04/Sep/15 05:48 Github user hyunsik commented on the pull request: https://github.com/apache/tajo/pull/726#issuecomment-137654265 Fixed test failures and rebased.

Hide

Permalink

ASF GitHub Bot added a comment - 07/Sep/15 06:04

Github user blrunner commented on a diff in the pull request:

https://github.com/apache/tajo/pull/726#discussion_r38835276

— Diff: tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java —
@@ -929,9 +933,16 @@ public static AlterTableDesc addOrDropPartition(String tableName, String[] colum
TUtil.putToNestedMap(OPERATION_CASTING_MAP, Type.FLOAT8, Type.TEXT, Type.TEXT);

TUtil.putToNestedMap(OPERATION_CASTING_MAP, Type.TEXT, Type.TIMESTAMP, Type.TIMESTAMP);
+ TUtil.putToNestedMap(OPERATION_CASTING_MAP, Type.TEXT, Type.TEXT, Type.TEXT);
— End diff –

We might need time type and date type.

Show

ASF GitHub Bot added a comment - 07/Sep/15 06:04 Github user blrunner commented on a diff in the pull request: https://github.com/apache/tajo/pull/726#discussion_r38835276 — Diff: tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java — @@ -929,9 +933,16 @@ public static AlterTableDesc addOrDropPartition(String tableName, String[] colum TUtil.putToNestedMap(OPERATION_CASTING_MAP, Type.FLOAT8, Type.TEXT, Type.TEXT); TUtil.putToNestedMap(OPERATION_CASTING_MAP, Type.TEXT, Type.TIMESTAMP, Type.TIMESTAMP); + TUtil.putToNestedMap(OPERATION_CASTING_MAP, Type.TEXT, Type.TEXT, Type.TEXT); — End diff – We might need time type and date type.

Hide

Permalink

ASF GitHub Bot added a comment - 07/Sep/15 06:10

Github user blrunner commented on a diff in the pull request:

https://github.com/apache/tajo/pull/726#discussion_r38835474

— Diff: tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestTablePartitions.java —
@@ -211,7 +208,7 @@ public final void testCreateColumnPartitionedTableWithSelectedColumns() throws E

TableDesc tableDesc = catalog.getTableDesc(DEFAULT_DATABASE_NAME, tableName);
verifyPartitionDirectoryFromCatalog(DEFAULT_DATABASE_NAME, tableName, new String[]

{"key"}

tableDesc.getStats().getNumRows());
+ tableDesc.getStats().getNumRows());
- - End diff –

This class seems to have unnecessary modifications.

Show

ASF GitHub Bot added a comment - 07/Sep/15 06:10 Github user blrunner commented on a diff in the pull request: https://github.com/apache/tajo/pull/726#discussion_r38835474 — Diff: tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestTablePartitions.java — @@ -211,7 +208,7 @@ public final void testCreateColumnPartitionedTableWithSelectedColumns() throws E TableDesc tableDesc = catalog.getTableDesc(DEFAULT_DATABASE_NAME, tableName); verifyPartitionDirectoryFromCatalog(DEFAULT_DATABASE_NAME, tableName, new String[] {"key"} , tableDesc.getStats().getNumRows()); + tableDesc.getStats().getNumRows()); End diff – This class seems to have unnecessary modifications.

Hide

Permalink

ASF GitHub Bot added a comment - 07/Sep/15 13:10

Github user hyunsik commented on the pull request:

https://github.com/apache/tajo/pull/726#issuecomment-138293860

I reflected the comments and rebased.

Show

ASF GitHub Bot added a comment - 07/Sep/15 13:10 Github user hyunsik commented on the pull request: https://github.com/apache/tajo/pull/726#issuecomment-138293860 I reflected the comments and rebased.

Hide

Permalink

ASF GitHub Bot added a comment - 08/Sep/15 06:19

Github user jihoonson commented on the pull request:

https://github.com/apache/tajo/pull/726#issuecomment-138447880

+1 ship it!

Show

ASF GitHub Bot added a comment - 08/Sep/15 06:19 Github user jihoonson commented on the pull request: https://github.com/apache/tajo/pull/726#issuecomment-138447880 +1 ship it!

Hide

Permalink

ASF GitHub Bot added a comment - 09/Sep/15 01:54

Github user asfgit closed the pull request at:

https://github.com/apache/tajo/pull/726

Show

ASF GitHub Bot added a comment - 09/Sep/15 01:54 Github user asfgit closed the pull request at: https://github.com/apache/tajo/pull/726

Hide

Permalink

Hudson added a comment - 09/Sep/15 01:55

FAILURE: Integrated in Tajo-master-CODEGEN-build #490 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/490/)
~~TAJO-1674~~: Validation of CTAS schema mismatch. (hyunsik: rev bdd8e76855eb0a79f22c67832edc5ce535a1a938)

CHANGES
tajo-core-tests/src/test/resources/queries/TestCreateTable/negative/type_mismatch.sql
tajo-plan/src/main/java/org/apache/tajo/plan/verifier/LogicalPlanVerifier.java
tajo-plan/src/main/java/org/apache/tajo/plan/verifier/SyntaxErrorUtil.java
tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java
tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestTablePartitions.java

Show

Hudson added a comment - 09/Sep/15 01:55 FAILURE: Integrated in Tajo-master-CODEGEN-build #490 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/490/ ) TAJO-1674 : Validation of CTAS schema mismatch. (hyunsik: rev bdd8e76855eb0a79f22c67832edc5ce535a1a938) CHANGES tajo-core-tests/src/test/resources/queries/TestCreateTable/negative/type_mismatch.sql tajo-plan/src/main/java/org/apache/tajo/plan/verifier/LogicalPlanVerifier.java tajo-plan/src/main/java/org/apache/tajo/plan/verifier/SyntaxErrorUtil.java tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestTablePartitions.java

Hide

Permalink

Hyunsik Choi added a comment - 09/Sep/15 02:10

I just committed it to master and branch-0.11.0 branches.

Show

Hyunsik Choi added a comment - 09/Sep/15 02:10 I just committed it to master and branch-0.11.0 branches.

Hide

Permalink

Hudson added a comment - 09/Sep/15 02:23

SUCCESS: Integrated in Tajo-master-build #848 (See https://builds.apache.org/job/Tajo-master-build/848/)
~~TAJO-1674~~: Validation of CTAS schema mismatch. (hyunsik: rev bdd8e76855eb0a79f22c67832edc5ce535a1a938)

tajo-plan/src/main/java/org/apache/tajo/plan/verifier/SyntaxErrorUtil.java
tajo-core-tests/src/test/resources/queries/TestCreateTable/negative/type_mismatch.sql
tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java
tajo-plan/src/main/java/org/apache/tajo/plan/verifier/LogicalPlanVerifier.java
CHANGES
tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestTablePartitions.java

Show

Hudson added a comment - 09/Sep/15 02:23 SUCCESS: Integrated in Tajo-master-build #848 (See https://builds.apache.org/job/Tajo-master-build/848/ ) TAJO-1674 : Validation of CTAS schema mismatch. (hyunsik: rev bdd8e76855eb0a79f22c67832edc5ce535a1a938) tajo-plan/src/main/java/org/apache/tajo/plan/verifier/SyntaxErrorUtil.java tajo-core-tests/src/test/resources/queries/TestCreateTable/negative/type_mismatch.sql tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java tajo-plan/src/main/java/org/apache/tajo/plan/verifier/LogicalPlanVerifier.java CHANGES tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestTablePartitions.java

Hide

Permalink

Hudson added a comment - 09/Sep/15 03:11

ABORTED: Integrated in Tajo-0.11.0-build #25 (See https://builds.apache.org/job/Tajo-0.11.0-build/25/)
~~TAJO-1674~~: Validation of CTAS schema mismatch. (hyunsik: rev 7b27c8ce9768aa56ed584757caa33f44d56a14ab)

CHANGES
tajo-plan/src/main/java/org/apache/tajo/plan/verifier/LogicalPlanVerifier.java
tajo-core-tests/src/test/resources/queries/TestCreateTable/negative/type_mismatch.sql
tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java
tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestTablePartitions.java
tajo-plan/src/main/java/org/apache/tajo/plan/verifier/SyntaxErrorUtil.java

Show

Hudson added a comment - 09/Sep/15 03:11 ABORTED: Integrated in Tajo-0.11.0-build #25 (See https://builds.apache.org/job/Tajo-0.11.0-build/25/ ) TAJO-1674 : Validation of CTAS schema mismatch. (hyunsik: rev 7b27c8ce9768aa56ed584757caa33f44d56a14ab) CHANGES tajo-plan/src/main/java/org/apache/tajo/plan/verifier/LogicalPlanVerifier.java tajo-core-tests/src/test/resources/queries/TestCreateTable/negative/type_mismatch.sql tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestTablePartitions.java tajo-plan/src/main/java/org/apache/tajo/plan/verifier/SyntaxErrorUtil.java

People

Assignee:

Hyunsik Choi

Reporter:

Jihoon Son

Votes:

0 Vote for this issue

Watchers:

3 Start watching this issue

Dates

Created:

06/Jul/15 07:55

Updated:

09/Sep/15 03:11

Resolved:

09/Sep/15 02:10

Development

Agile

View on Board