Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-1690

Create table using text file fails in HiveCatalogStore.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: 0.11.0
    • Component/s: Catalog
    • Labels:
      None

      Description

      Create table using text file fails in HiveCatalogStore as follows.

      • TSQL
        default> create external table supplier (S_SUPPKEY bigint, S_NAME text, S_ADDRESS text, S_NATIONKEY bigint, S_PHONE text, S_ACCTBAL double, S_COMMENT text) using text with ('textfile.delimiter'='|') location 'hdfs://localhost:9010/tpch/supplier';
        ERROR: Cannot create table "default.supplier".
        
      • TajoMaster log
        2015-07-18 01:40:12,055 ERROR org.apache.tajo.catalog.CatalogServer: org.apache.tajo.common.exception.NotImplementedException: text
        org.apache.tajo.catalog.exception.CatalogException: org.apache.tajo.common.exception.NotImplementedException: text
        	at org.apache.tajo.catalog.store.HiveCatalogStore.createTable(HiveCatalogStore.java:553)
        	at org.apache.tajo.catalog.CatalogServer$CatalogProtocolHandler.createTable(CatalogServer.java:602)
        	at org.apache.tajo.catalog.AbstractCatalogClient.createTable(AbstractCatalogClient.java:406)
        	at org.apache.tajo.master.exec.DDLExecutor.createTable(DDLExecutor.java:275)
        	at org.apache.tajo.master.exec.DDLExecutor.createTable(DDLExecutor.java:211)
        	at org.apache.tajo.master.exec.DDLExecutor.execute(DDLExecutor.java:91)
        	at org.apache.tajo.master.exec.QueryExecutor.execute(QueryExecutor.java:107)
        

      It seems that HiveCatalogStore just support csv file format. We need to support text file format in HiveCatalogStore.

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user blrunner opened a pull request:

          https://github.com/apache/tajo/pull/635

          TAJO-1690: Create table using text file fails in HiveCatalogStore.

          I added text file format to HiveCatalogStore.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/blrunner/tajo TAJO-1690

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/tajo/pull/635.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #635


          commit 33da561c0dbd06dcfbe7772f113c62921e47e0d3
          Author: JaeHwa Jung <blrunner@apache.org>
          Date: 2015-07-17T17:00:09Z

          TAJO-1690: Create table using text file fails in HiveCatalogStore.


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user blrunner opened a pull request: https://github.com/apache/tajo/pull/635 TAJO-1690 : Create table using text file fails in HiveCatalogStore. I added text file format to HiveCatalogStore. You can merge this pull request into a Git repository by running: $ git pull https://github.com/blrunner/tajo TAJO-1690 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/tajo/pull/635.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #635 commit 33da561c0dbd06dcfbe7772f113c62921e47e0d3 Author: JaeHwa Jung <blrunner@apache.org> Date: 2015-07-17T17:00:09Z TAJO-1690 : Create table using text file fails in HiveCatalogStore.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/635#discussion_r34965599

          — Diff: tajo-catalog/tajo-catalog-drivers/tajo-hive/src/test/java/org/apache/tajo/catalog/store/TestHiveCatalogStore.java —
          @@ -85,34 +86,36 @@ public static void tearDown() throws IOException {

          @Test
          public void testTableUsingTextFile() throws Exception {

          • TableMeta meta = new TableMeta("CSV", new KeyValueSet());
            -
          • org.apache.tajo.catalog.Schema schema = new org.apache.tajo.catalog.Schema();
          • schema.addColumn("c_custkey", TajoDataTypes.Type.INT4);
          • schema.addColumn("c_name", TajoDataTypes.Type.TEXT);
          • schema.addColumn("c_address", TajoDataTypes.Type.TEXT);
          • schema.addColumn("c_nationkey", TajoDataTypes.Type.INT4);
          • schema.addColumn("c_phone", TajoDataTypes.Type.TEXT);
          • schema.addColumn("c_acctbal", TajoDataTypes.Type.FLOAT8);
          • schema.addColumn("c_mktsegment", TajoDataTypes.Type.TEXT);
          • schema.addColumn("c_comment", TajoDataTypes.Type.TEXT);
            -
          • TableDesc table = new TableDesc(CatalogUtil.buildFQName(DB_NAME, CUSTOMER), schema, meta,
            + for (String storage: storages) {
              • End diff –

          I don't think that we need all unit tests for each storage type. You just add OR condition for TEXT and CSV and all other codes are the same. You need to add an unit test about your change.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/635#discussion_r34965599 — Diff: tajo-catalog/tajo-catalog-drivers/tajo-hive/src/test/java/org/apache/tajo/catalog/store/TestHiveCatalogStore.java — @@ -85,34 +86,36 @@ public static void tearDown() throws IOException { @Test public void testTableUsingTextFile() throws Exception { TableMeta meta = new TableMeta("CSV", new KeyValueSet()); - org.apache.tajo.catalog.Schema schema = new org.apache.tajo.catalog.Schema(); schema.addColumn("c_custkey", TajoDataTypes.Type.INT4); schema.addColumn("c_name", TajoDataTypes.Type.TEXT); schema.addColumn("c_address", TajoDataTypes.Type.TEXT); schema.addColumn("c_nationkey", TajoDataTypes.Type.INT4); schema.addColumn("c_phone", TajoDataTypes.Type.TEXT); schema.addColumn("c_acctbal", TajoDataTypes.Type.FLOAT8); schema.addColumn("c_mktsegment", TajoDataTypes.Type.TEXT); schema.addColumn("c_comment", TajoDataTypes.Type.TEXT); - TableDesc table = new TableDesc(CatalogUtil.buildFQName(DB_NAME, CUSTOMER), schema, meta, + for (String storage: storages) { End diff – I don't think that we need all unit tests for each storage type. You just add OR condition for TEXT and CSV and all other codes are the same. You need to add an unit test about your change.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user blrunner commented on the pull request:

          https://github.com/apache/tajo/pull/635#issuecomment-122746746

          Thanks @hyunsik and @jinossy

          I've recovered existing test cases and add just a test case for this issue.

          Show
          githubbot ASF GitHub Bot added a comment - Github user blrunner commented on the pull request: https://github.com/apache/tajo/pull/635#issuecomment-122746746 Thanks @hyunsik and @jinossy I've recovered existing test cases and add just a test case for this issue.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on the pull request:

          https://github.com/apache/tajo/pull/635#issuecomment-123555182

          The change in this patch seems to be duplicated to #638.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on the pull request: https://github.com/apache/tajo/pull/635#issuecomment-123555182 The change in this patch seems to be duplicated to #638.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user blrunner commented on the pull request:

          https://github.com/apache/tajo/pull/635#issuecomment-123557995

          @hyunsik

          OK. I'll close this PR.

          Show
          githubbot ASF GitHub Bot added a comment - Github user blrunner commented on the pull request: https://github.com/apache/tajo/pull/635#issuecomment-123557995 @hyunsik OK. I'll close this PR.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user blrunner closed the pull request at:

          https://github.com/apache/tajo/pull/635

          Show
          githubbot ASF GitHub Bot added a comment - Github user blrunner closed the pull request at: https://github.com/apache/tajo/pull/635

            People

            • Assignee:
              blrunner Jaehwa Jung
              Reporter:
              blrunner Jaehwa Jung
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development