Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-1922

hql query throws "RuntimeException: Unsupported dataType" if struct field of a table has a column with underscore in name

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.0.0
    • 1.0.1, 1.1.0
    • SQL
    • None

    Description

      If table A has a struct field A_strct <a_col: string>, when doing an hql query like "select", "RuntimeException: Unsupported dataType" is thrown.

      Running a query wiht "sbt/sbt hive/console":

      scala> hql("SELECT utc_time, pkg FROM pkg_table where year=2014 and month=1 limit 10").collect().foreach(x => println(x(1)))
      

      Console output:

      14/05/25 19:50:27 INFO parse.ParseDriver: Parsing command: SELECT utc_time, pkg FROM pkg_table where year=2014 and month=1 limit 10
      14/05/25 19:50:27 INFO parse.ParseDriver: Parse Completed
      14/05/25 19:50:28 INFO analysis.Analyzer: Max iterations (2) reached for batch MultiInstanceRelations
      14/05/25 19:50:28 INFO analysis.Analyzer: Max iterations (2) reached for batch CaseInsensitiveAttributeReferences
      14/05/25 19:50:28 INFO hive.metastore: Trying to connect to metastore with URI thrift://xxxxx
      14/05/25 19:50:28 INFO hive.metastore: Waiting 1 seconds before next connection attempt.
      14/05/25 19:50:29 INFO hive.metastore: Connected to metastore.
      java.lang.RuntimeException: Unsupported dataType: struct<adv_id:string,u_lat:double,u_lon:double>
      	at scala.sys.package$.error(package.scala:27)
      	at org.apache.spark.sql.hive.HiveMetastoreTypes$.toDataType(HiveMetastoreCatalog.scala:219)
      	at org.apache.spark.sql.hive.MetastoreRelation$SchemaAttribute.toAttribute(HiveMetastoreCatalog.scala:273)
      	at org.apache.spark.sql.hive.MetastoreRelation$$anonfun$8.apply(HiveMetastoreCatalog.scala:283)
      	at org.apache.spark.sql.hive.MetastoreRelation$$anonfun$8.apply(HiveMetastoreCatalog.scala:283)
      	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
      

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            llai llai
            llai llai
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment