[SPARK-22431] Creating Permanent view with illegal type

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.2.0
• Fix Version/s: 2.3.0
• Component/s: SQL
• Target Version/s: 2.3.0
• Assignee: Sunitha Kambhampati
• Reporter: Herman van Hövell
• Created: 02/Nov/17 22:15
• Updated: 28/Nov/17 21:09
• Resolved: 28/Nov/17 21:09

Description

It is possible in Spark SQL to create a permanent view that uses an nested field with an illegal name.

For example if we create the following view:

create view x as select struct('a' as `$q`, 1 as b) q

A simple select fails with the following exception:

select * from x;

org.apache.spark.SparkException: Cannot recognize hive type string: struct<$q:string,b:int>
  at org.apache.spark.sql.hive.client.HiveClientImpl$.fromHiveColumn(HiveClientImpl.scala:812)
  at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$getTableOption$1$$anonfun$apply$11$$anonfun$7.apply(HiveClientImpl.scala:378)
  at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$getTableOption$1$$anonfun$apply$11$$anonfun$7.apply(HiveClientImpl.scala:378)
...

Dropping the view isn't possible either:

drop view x;

org.apache.spark.SparkException: Cannot recognize hive type string: struct<$q:string,b:int>
  at org.apache.spark.sql.hive.client.HiveClientImpl$.fromHiveColumn(HiveClientImpl.scala:812)
  at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$getTableOption$1$$anonfun$apply$11$$anonfun$7.apply(HiveClientImpl.scala:378)
  at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$getTableOption$1$$anonfun$apply$11$$anonfun$7.apply(HiveClientImpl.scala:378)
...

Issue Links

• links to: [Github] Pull Request #19747 (skambha)