Uploaded image for project: 'Phoenix'
  1. Phoenix
  2. PHOENIX-2336

Queries with small case column-names return empty result-set when working with Spark Datasource Plugin

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 4.6.0
    • Fix Version/s: 4.9.0, 4.8.1
    • Labels:

      Description

      Hi,

      The Spark DataFrame filter operation returns empty result-set when column-name is in the smaller case. Example below:
      DataFrame df = sqlContext.read().format("org.apache.phoenix.spark").options(params).load();
      df.filter("\"col1\" = '5.0'").show();

      Result:
      ---------

      ID col1 c1 d2 d3 d4

      ---------+
      ---------+

      Whereas the table actually has some rows matching the filter condition. And if double quotes are removed from around the column name i.e. df.filter("col1 = '5.0'").show(); , a ColumnNotFoundException is thrown:
      Exception in thread "main" java.lang.RuntimeException: org.apache.phoenix.schema.ColumnNotFoundException: ERROR 504 (42703): Undefined column. columnName=D1
      at org.apache.phoenix.mapreduce.PhoenixInputFormat.getQueryPlan(PhoenixInputFormat.java:125)
      at org.apache.phoenix.mapreduce.PhoenixInputFormat.getSplits(PhoenixInputFormat.java:80)
      at org.apache.spark.rdd.NewHadoopRDD.getPartitions(NewHadoopRDD.scala:95)
      at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
      at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
      at scala.Option.getOrElse(Option.scala:120)

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                kalyanhadoop Kalyan
                Reporter:
                snalapure@dataken.net Suhas Nalapure
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: