[PHOENIX-2336] Queries with small case column-names return empty result-set when working with Spark Datasource Plugin

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.6.0
• Fix Version/s: 4.9.0, 4.8.1
• Labels: verify
• Assignee: Kalyan
• Reporter: Suhas Nalapure
• Created: 19/Oct/15 07:40
• Updated: 28/Sep/16 05:15
• Resolved: 10/Sep/16 20:10

Description

Hi,

The Spark DataFrame filter operation returns empty result-set when column-name is in the smaller case. Example below:
DataFrame df = sqlContext.read().format("org.apache.phoenix.spark").options(params).load();
df.filter("\"col1\" = '5.0'").show();

Result:
---------

ID

col1

c1

d2

d3

d4

---------+
---------+

Whereas the table actually has some rows matching the filter condition. And if double quotes are removed from around the column name i.e. df.filter("col1 = '5.0'").show(); , a ColumnNotFoundException is thrown:
Exception in thread "main" java.lang.RuntimeException: org.apache.phoenix.schema.ColumnNotFoundException: ERROR 504 (42703): Undefined column. columnName=D1
at org.apache.phoenix.mapreduce.PhoenixInputFormat.getQueryPlan(PhoenixInputFormat.java:125)
at org.apache.phoenix.mapreduce.PhoenixInputFormat.getSplits(PhoenixInputFormat.java:80)
at org.apache.spark.rdd.NewHadoopRDD.getPartitions(NewHadoopRDD.scala:95)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
at scala.Option.getOrElse(Option.scala:120)

Issue Links

• is related to: PHOENIX-2290 Spark Phoenix cannot recognize Phoenix view fields