Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-1922

hql query throws "RuntimeException: Unsupported dataType" if struct field of a table has a column with underscore in name

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.0
    • Fix Version/s: 1.0.1, 1.1.0
    • Component/s: SQL
    • Labels:
      None

      Description

      If table A has a struct field A_strct <a_col: string>, when doing an hql query like "select", "RuntimeException: Unsupported dataType" is thrown.

      Running a query wiht "sbt/sbt hive/console":

      scala> hql("SELECT utc_time, pkg FROM pkg_table where year=2014 and month=1 limit 10").collect().foreach(x => println(x(1)))
      

      Console output:

      14/05/25 19:50:27 INFO parse.ParseDriver: Parsing command: SELECT utc_time, pkg FROM pkg_table where year=2014 and month=1 limit 10
      14/05/25 19:50:27 INFO parse.ParseDriver: Parse Completed
      14/05/25 19:50:28 INFO analysis.Analyzer: Max iterations (2) reached for batch MultiInstanceRelations
      14/05/25 19:50:28 INFO analysis.Analyzer: Max iterations (2) reached for batch CaseInsensitiveAttributeReferences
      14/05/25 19:50:28 INFO hive.metastore: Trying to connect to metastore with URI thrift://xxxxx
      14/05/25 19:50:28 INFO hive.metastore: Waiting 1 seconds before next connection attempt.
      14/05/25 19:50:29 INFO hive.metastore: Connected to metastore.
      java.lang.RuntimeException: Unsupported dataType: struct<adv_id:string,u_lat:double,u_lon:double>
      	at scala.sys.package$.error(package.scala:27)
      	at org.apache.spark.sql.hive.HiveMetastoreTypes$.toDataType(HiveMetastoreCatalog.scala:219)
      	at org.apache.spark.sql.hive.MetastoreRelation$SchemaAttribute.toAttribute(HiveMetastoreCatalog.scala:273)
      	at org.apache.spark.sql.hive.MetastoreRelation$$anonfun$8.apply(HiveMetastoreCatalog.scala:283)
      	at org.apache.spark.sql.hive.MetastoreRelation$$anonfun$8.apply(HiveMetastoreCatalog.scala:283)
      	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
      

        Attachments

          Activity

            People

            • Assignee:
              llai llai
              Reporter:
              llai llai
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: