[SPARK-18355] Spark SQL fails to read data from a ORC hive table that has a new column added to it

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.6.2, 2.1.0, 2.2.0
• Fix Version/s: 2.2.1, 2.3.0
• Component/s: SQL

Environment

Centos6

• Assignee: Dongjoon Hyun
• Reporter: Sandeep Nemuri
• Created: 08/Nov/16 09:45
• Updated: 03/Nov/17 04:44
• Resolved: 13/Oct/17 15:25

Description

PROBLEM:

Spark SQL fails to read data from a ORC hive table that has a new column added to it.

Below is the exception:

scala> sqlContext.sql("select click_id,search_id from testorc").show
16/11/03 22:17:53 INFO ParseDriver: Parsing command: select click_id,search_id from testorc
16/11/03 22:17:54 INFO ParseDriver: Parse Completed
java.lang.AssertionError: assertion failed
	at scala.Predef$.assert(Predef.scala:165)
	at org.apache.spark.sql.execution.datasources.LogicalRelation$$anonfun$1.apply(LogicalRelation.scala:39)
	at org.apache.spark.sql.execution.datasources.LogicalRelation$$anonfun$1.apply(LogicalRelation.scala:38)
	at scala.Option.map(Option.scala:145)
	at org.apache.spark.sql.execution.datasources.LogicalRelation.<init>(LogicalRelation.scala:38)
	at org.apache.spark.sql.execution.datasources.LogicalRelation.copy(LogicalRelation.scala:31)
	at org.apache.spark.sql.hive.HiveMetastoreCatalog.org$apache$spark$sql$hive$HiveMetastoreCatalog$$convertToOrcRelation(HiveMetastoreCatalog.scala:588)
	at org.apache.spark.sql.hive.HiveMetastoreCatalog$OrcConversions$$anonfun$apply$2.applyOrElse(HiveMetastoreCatalog.scala:647)
	at org.apache.spark.sql.hive.HiveMetastoreCatalog$OrcConversions$$anonfun$apply$2.applyOrElse(HiveMetastoreCatalog.scala:643)
	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:335)
	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:335)
	at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:69)
	at org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:334)
	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$5.apply(TreeNode.scala:332)
	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$5.apply(TreeNode.scala:332)
	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:281)
	at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)

STEPS TO SIMULATE THIS ISSUE:

1) Create table in hive.

CREATE TABLE `testorc`( 
`click_id` string, 
`search_id` string, 
`uid` bigint)
PARTITIONED BY ( 
`ts` string, 
`hour` string) 
STORED AS ORC; 

2) Load data into table :

INSERT INTO TABLE testorc PARTITION (ts = '98765',hour = '01' ) VALUES (12,2,12345);

3) Select through spark shell (This works)

sqlContext.sql("select click_id,search_id from testorc").show

4) Now add column to hive table

ALTER TABLE testorc ADD COLUMNS (dummy string);

5) Now again select from spark shell

scala> sqlContext.sql("select click_id,search_id from testorc").show
16/11/03 22:17:53 INFO ParseDriver: Parsing command: select click_id,search_id from testorc
16/11/03 22:17:54 INFO ParseDriver: Parse Completed
java.lang.AssertionError: assertion failed
	at scala.Predef$.assert(Predef.scala:165)
	at org.apache.spark.sql.execution.datasources.LogicalRelation$$anonfun$1.apply(LogicalRelation.scala:39)
	at org.apache.spark.sql.execution.datasources.LogicalRelation$$anonfun$1.apply(LogicalRelation.scala:38)
	at scala.Option.map(Option.scala:145)
	at org.apache.spark.sql.execution.datasources.LogicalRelation.<init>(LogicalRelation.scala:38)
	at org.apache.spark.sql.execution.datasources.LogicalRelation.copy(LogicalRelation.scala:31)
	at org.apache.spark.sql.hive.HiveMetastoreCatalog.org$apache$spark$sql$hive$HiveMetastoreCatalog$$convertToOrcRelation(HiveMetastoreCatalog.scala:588)
	at org.apache.spark.sql.hive.HiveMetastoreCatalog$OrcConversions$$anonfun$apply$2.applyOrElse(HiveMetastoreCatalog.scala:647)
	at org.apache.spark.sql.hive.HiveMetastoreCatalog$OrcConversions$$anonfun$apply$2.applyOrElse(HiveMetastoreCatalog.scala:643)
	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:335)
	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:335)
	at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:69)
	at org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:334)
	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$5.apply(TreeNode.scala:332)
	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$5.apply(TreeNode.scala:332)
	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:281)
	at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
	at scala.collection.Iterator$class.foreach(Iterator.scala:727)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
	at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
	at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)

Issue Links

• blocks: SPARK-20901 Feature parity for ORC with Parquet
• is duplicated by: SPARK-21686 spark.sql.hive.convertMetastoreOrc is causing NullPointerException while reading ORC tables
• relates to: SPARK-16628 OrcConversions should not convert an ORC table represented by MetastoreRelation to HadoopFsRelation if metastore schema does not match schema stored in ORC files
• links to: [Github] Pull Request #19470 (dongjoon-hyun)