Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-2820

Impala does not support reserved words in complex types

    XMLWordPrintableJSON

    Details

      Description

      I have been testing CDH5.5.0 and have noted that Impala does not like reserved words as field names in complex types. This seems strange as reserved words can be used as column names for ordinary columns. Hive does not impose the same restriction. Reserved words can be back-ticked where needed. I am using Hive to create Parquet files with complex types.

      I discussed this in this in the cloudera community in http://community.cloudera.com/t5/Interactive-Short-cycle-SQL/Impala-does-not-not-support-reserved-words-in-complex-types/m-p/35937#U35937.

      Alex Behm replied with
      "I confirm that this is a problem in Impala 2.3. It stems from the fact that the type is stored as a string inside the Hive Metastore, and upon table loading, Impala parses the string into an internal representation. However, the types stored inside the Metastore are not quoted, so Impala fails to parse the reserved words."

      Sample to reproduce issue and error message below. In the case below the word 'replace' is reserved.

      In Hive:

      CREATE EXTERNAL TABLE MyTable (
        device_id STRING,
        added struct<name:string,version_name:string,version_code:int,`replace`:boolean>
      )
      STORED AS PARQUET
      LOCATION '/tmp/impala/mytable'; 
      

      In Hive:

      INSERT OVERWRITE TABLE MyTable
      SELECT  
        device_id,
        payload AS added
      FROM Added where import_id = 106000;
      

      In Impala:

      SELECT * FROM MyTable limit 10;
      

      Output:

      AnalysisException: Failed to load metadata for table: 'mytable' CAUSED BY: TableLoadingException: Unsupported type 'struct<name:string,version_name:string,version_code:int,replace:boolean>' in column 'added' of table 'mytable'  
      
      
      I0107 15:56:01.251721 21006 Frontend.java:818] analyze query SELECT * FROM MyTable limit 10
      E0107 15:56:01.252320 21006 Analyzer.java:2212] Failed to load metadata for table: mytable
      Unsupported type 'struct<name:string,version_name:string,version_code:int,replace:boolean>' in column 'added' of table 'mytable'
      I0107 15:56:01.252908 21006 jni-util.cc:177] com.cloudera.impala.common.AnalysisException: Failed to load metadata for table: 'MyTable'
              at com.cloudera.impala.analysis.TableRef.analyze(TableRef.java:180)
              at com.cloudera.impala.analysis.Analyzer.resolveTableRef(Analyzer.java:512)
              at com.cloudera.impala.analysis.SelectStmt.analyze(SelectStmt.java:155)
              at com.cloudera.impala.analysis.AnalysisContext.analyze(AnalysisContext.java:342)
              at com.cloudera.impala.analysis.AnalysisContext.analyze(AnalysisContext.java:317)
              at com.cloudera.impala.service.Frontend.analyzeStmt(Frontend.java:827)
              at com.cloudera.impala.service.Frontend.createExecRequest(Frontend.java:856)
              at com.cloudera.impala.service.JniFrontend.createExecRequest(JniFrontend.java:147)
      Caused by: com.cloudera.impala.catalog.TableLoadingException: Unsupported type 'struct<name:string,version_name:string,version_code:int,replace:boolean>' in column 'added' of table 'mytable'
              at com.cloudera.impala.catalog.IncompleteTable.loadFromThrift(IncompleteTable.java:111)
              at com.cloudera.impala.catalog.Table.fromThrift(Table.java:240)
              at com.cloudera.impala.catalog.ImpaladCatalog.addTable(ImpaladCatalog.java:357)
              at com.cloudera.impala.catalog.ImpaladCatalog.addCatalogObject(ImpaladCatalog.java:246)
              at com.cloudera.impala.catalog.ImpaladCatalog.updateCatalog(ImpaladCatalog.java:132)
              at com.cloudera.impala.service.Frontend.updateCatalogCache(Frontend.java:223)
              at com.cloudera.impala.service.JniFrontend.updateCatalogCache(JniFrontend.java:164)
              at ========.<Remote stack trace on catalogd>: com.cloudera.impala.catalog.TableLoadingException: Unsupported type 'struct<name:string,version_name:string,version_code:int,replace:boolean>' in column 'added' of table 'mytable'
              at com.cloudera.impala.catalog.Table.parseColumnType(Table.java:331)
              at com.cloudera.impala.catalog.HdfsTable.addColumnsFromFieldSchemas(HdfsTable.java:571)
              at com.cloudera.impala.catalog.HdfsTable.load(HdfsTable.java:1073)
              at com.cloudera.impala.catalog.TableLoader.load(TableLoader.java:84)
              at com.cloudera.impala.catalog.TableLoadingMgr$2.call(TableLoadingMgr.java:232)
              at com.cloudera.impala.catalog.TableLoadingMgr$2.call(TableLoadingMgr.java:229)
              at java.util.concurrent.FutureTask.run(FutureTask.java:262)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
              at java.lang.Thread.run(Thread.java:745)
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                alex.behm Alexander Behm
                Reporter:
                Pettax Petter von Dolwitz
              • Votes:
                1 Vote for this issue
                Watchers:
                11 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: