[SPARK-4959] Attributes are case sensitive when using a select query from a projection

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.2.0
• Fix Version/s: 1.2.1, 1.3.0
• Component/s: SQL
• Labels: backport-needed
• Target Version/s: 1.2.1, 1.3.0
• Assignee: Cheng Hao
• Reporter: Andy Konwinski
• Shepherd: Michael Armbrust
• Created: 24/Dec/14 22:37
• Updated: 30/Jan/15 00:44
• Resolved: 30/Jan/15 00:44

Description

Per marmbrus, see this line of code, where we should be using an attribute map
https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala#L147

To reproduce, i ran the following in the Spark shell:

import sqlContext._
sql("drop table if exists test")
sql("create table test (col1 string)")
sql("""insert into table test select "hi" from prejoined limit 1""")
val projection = "col1".attr.as(Symbol("CaseSensitiveColName")) :: "col1".attr.as(Symbol("CaseSensitiveColName2")) :: Nil
sqlContext.table("test").select(projection:_*).registerTempTable("test2")

# This succeeds.
sql("select CaseSensitiveColName from test2").first()

# This fails with java.util.NoSuchElementException: key not found: casesensitivecolname#23046
sql("select casesensitivecolname from test2").first()

The full stack trace printed for the final command that is failing:

java.util.NoSuchElementException: key not found: casesensitivecolname#23046
	at scala.collection.MapLike$class.default(MapLike.scala:228)
	at org.apache.spark.sql.catalyst.expressions.AttributeMap.default(AttributeMap.scala:29)
	at scala.collection.MapLike$class.apply(MapLike.scala:141)
	at org.apache.spark.sql.catalyst.expressions.AttributeMap.apply(AttributeMap.scala:29)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
	at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
	at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
	at scala.collection.AbstractTraversable.map(Traversable.scala:105)
	at org.apache.spark.sql.hive.execution.HiveTableScan.<init>(HiveTableScan.scala:57)
	at org.apache.spark.sql.hive.HiveStrategies$HiveTableScans$$anonfun$14.apply(HiveStrategies.scala:221)
	at org.apache.spark.sql.hive.HiveStrategies$HiveTableScans$$anonfun$14.apply(HiveStrategies.scala:221)
	at org.apache.spark.sql.SQLContext$SparkPlanner.pruneFilterProject(SQLContext.scala:378)
	at org.apache.spark.sql.hive.HiveStrategies$HiveTableScans$.apply(HiveStrategies.scala:217)
	at org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$1.apply(QueryPlanner.scala:58)
	at org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$1.apply(QueryPlanner.scala:58)
	at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
	at org.apache.spark.sql.catalyst.planning.QueryPlanner.apply(QueryPlanner.scala:59)
	at org.apache.spark.sql.catalyst.planning.QueryPlanner.planLater(QueryPlanner.scala:54)
	at org.apache.spark.sql.execution.SparkStrategies$BasicOperators$.apply(SparkStrategies.scala:285)
	at org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$1.apply(QueryPlanner.scala:58)
	at org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$1.apply(QueryPlanner.scala:58)
	at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
	at org.apache.spark.sql.catalyst.planning.QueryPlanner.apply(QueryPlanner.scala:59)
	at org.apache.spark.sql.SQLContext$QueryExecution.sparkPlan$lzycompute(SQLContext.scala:418)
	at org.apache.spark.sql.SQLContext$QueryExecution.sparkPlan(SQLContext.scala:416)
	at org.apache.spark.sql.SQLContext$QueryExecution.executedPlan$lzycompute(SQLContext.scala:422)
	at org.apache.spark.sql.SQLContext$QueryExecution.executedPlan(SQLContext.scala:422)
	at org.apache.spark.sql.SchemaRDD.collect(SchemaRDD.scala:444)
	at org.apache.spark.sql.SchemaRDD.take(SchemaRDD.scala:446)
	at org.apache.spark.sql.SchemaRDD.take(SchemaRDD.scala:108)
	at org.apache.spark.rdd.RDD.first(RDD.scala:1093)

Issue Links

• links to: [Github] Pull Request #3796 (chenghao-intel)
• links to: [Github] Pull Request #4013 (chenghao-intel)