Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-2821

error when trying to drop or compute stats of table with a nested field in a struct column named `timestamp`

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: Impala 2.3.0
    • Fix Version/s: None
    • Component/s: Frontend
    • Labels:
      None

      Description

      Issue: quoted identifiers don't work in structs in external tables. I can create a table but can't drop it or compute stats. The error is

      ERROR: AnalysisException: Unsupported type 'struct<timestamp:bigint>' in column 'header' of table 'test3'
      

      CDH 5.5.1 with Impala v2.3.0-cdh5.5.1 (73bf5bc) built on Wed Dec 2 10:39:33 PST 2015

      How to reproduce: (spark-shell on one of datanodes)
      This works

      [localhost:21000] > create external table test4 (header struct<`ts`: bigint>) stored as parquet;
      Query: create external table test4 (header struct<`ts`: bigint>) stored as parquet
      Fetched 0 row(s) in 0.08s
      
      [localhost:21000] > compute stats test4;
      Query: compute stats test4
      +-----------------------------------------+
      | summary                                 |
      +-----------------------------------------+
      | Updated 1 partition(s) and 0 column(s). |
      +-----------------------------------------+
      Fetched 1 row(s) in 6.68s
      
      [localhost:21000] > drop table test4;
      Query: drop table test4
      

      This fails

      [localhost:21000] > create external table test3 (header struct<`timestamp`: bigint>) stored as parquet;
      Query: create external table test3 (header struct<`timestamp`: bigint>) stored as parquet
      Fetched 0 row(s) in 0.06s
      
      [localhost:21000] > drop table test3;
      Query: drop table test3
      ERROR: AnalysisException: Unsupported type 'struct<timestamp:bigint>' in column 'header' of table 'test3'
      CAUSED BY: TableLoadingException: Unsupported type 'struct<timestamp:bigint>' in column 'header' of table 'test3'
      

      Compute stats fails too

      [localhost:21000] > compute stats test3;
      Query: compute stats test3
      ERROR: AnalysisException: Unsupported type 'struct<timestamp:bigint>' in column 'header' of table 'test3'
      CAUSED BY: TableLoadingException: Unsupported type 'struct<timestamp:bigint>' in column 'header' of table 'test3'
      

      Can not query this table

      [localhost:21000] > select header.`timestamp` from test3;
      Query: select header.`timestamp` from test3
      ERROR: AnalysisException: Failed to load metadata for table: 'test3'
      CAUSED BY: TableLoadingException: Unsupported type 'struct<timestamp:bigint>' in column 'header' of table 'test3'
      

      Top level columns support quoted identifiers

      [localhost:21000] > create external table test2 (`timestamp` bigint) stored as parquet;
      Query: create external table test2 (`timestamp` bigint) stored as parquet
      Fetched 0 row(s) in 0.09s
      
      [localhost:21000] > compute stats test2;
      Query: compute stats test2
      +-----------------------------------------+
      | summary                                 |
      +-----------------------------------------+
      | Updated 1 partition(s) and 1 column(s). |
      +-----------------------------------------+
      Fetched 1 row(s) in 4.54s
      
      [localhost:21000] > drop table test2;
      Query: drop table test3
      

      Workaround found for table removal: drop the table from hive metastore in HUE.

      No workaround found for compute stats or querying the table

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                epishkin oleksii iepishkin
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: