[SPARK-1922] hql query throws "RuntimeException: Unsupported dataType" if struct field of a table has a column with underscore in name

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 1.0.1, 1.1.0
• Component/s: SQL
• Assignee: llai
• Reporter: llai
• Created: 25/May/14 12:16
• Updated: 27/May/14 23:09
• Resolved: 27/May/14 23:09

Description

If table A has a struct field A_strct <a_col: string>, when doing an hql query like "select", "RuntimeException: Unsupported dataType" is thrown.

Running a query wiht "sbt/sbt hive/console":

scala> hql("SELECT utc_time, pkg FROM pkg_table where year=2014 and month=1 limit 10").collect().foreach(x => println(x(1)))

Console output:

14/05/25 19:50:27 INFO parse.ParseDriver: Parsing command: SELECT utc_time, pkg FROM pkg_table where year=2014 and month=1 limit 10
14/05/25 19:50:27 INFO parse.ParseDriver: Parse Completed
14/05/25 19:50:28 INFO analysis.Analyzer: Max iterations (2) reached for batch MultiInstanceRelations
14/05/25 19:50:28 INFO analysis.Analyzer: Max iterations (2) reached for batch CaseInsensitiveAttributeReferences
14/05/25 19:50:28 INFO hive.metastore: Trying to connect to metastore with URI thrift://xxxxx
14/05/25 19:50:28 INFO hive.metastore: Waiting 1 seconds before next connection attempt.
14/05/25 19:50:29 INFO hive.metastore: Connected to metastore.
java.lang.RuntimeException: Unsupported dataType: struct<adv_id:string,u_lat:double,u_lon:double>
	at scala.sys.package$.error(package.scala:27)
	at org.apache.spark.sql.hive.HiveMetastoreTypes$.toDataType(HiveMetastoreCatalog.scala:219)
	at org.apache.spark.sql.hive.MetastoreRelation$SchemaAttribute.toAttribute(HiveMetastoreCatalog.scala:273)
	at org.apache.spark.sql.hive.MetastoreRelation$$anonfun$8.apply(HiveMetastoreCatalog.scala:283)
	at org.apache.spark.sql.hive.MetastoreRelation$$anonfun$8.apply(HiveMetastoreCatalog.scala:283)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)