[SPARK-41990] Filtering by composite field name like `field name` doesn't work with pushDownPredicate = true

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0, 3.3.1
• Fix Version/s: 3.4.0
• Component/s: SQL
• Assignee: Huaxin Gao
• Reporter: Marina Krasilnikova
• Created: 11/Jan/23 19:08
• Updated: 13/Feb/23 06:46
• Resolved: 15/Jan/23 06:53

Description

Suppose we have some table in postgresql with field `Last Name` The following code results in error

Dataset<Row> dataset = sparkSession.read()
.format("jdbc")
.option("url", myUrl)
.option("dbtable", "myTable")
.option("user", "myUser")
.option("password", "muPassword")
.load();

dataset.where("`Last Name`='Tessel'").show();    //error

 

 

Exception in thread "main" org.apache.spark.sql.catalyst.parser.ParseException: 
Syntax error at or near 'Name': extra input 'Name'(line 1, pos 5)

== SQL ==
Last Name
-----^^^

    at org.apache.spark.sql.catalyst.parser.ParseException.withCommand(ParseDriver.scala:306)
    at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:143)
    at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parseMultipartIdentifier(ParseDriver.scala:67)
    at org.apache.spark.sql.connector.expressions.LogicalExpressions$.parseReference(expressions.scala:40)
    at org.apache.spark.sql.connector.expressions.FieldReference$.apply(expressions.scala:368)
    at org.apache.spark.sql.sources.IsNotNull.toV2(filters.scala:262)
    at org.apache.spark.sql.execution.datasources.jdbc.JDBCRelation.$anonfun$unhandledFilters$1(JDBCRelation.scala:278)
    at org.apache.spark.sql.execution.datasources.jdbc.JDBCRelation.$anonfun$unhandledFilters$1$adapted(JDBCRelation.scala:278)

 

But if we set pushDownPredicate to false everything works fine.

Issue Links

• is duplicated by: SPARK-42193 dataframe API filter criteria throwing ParseException when reading a JDBC column name with special characters
• links to: [Github] Pull Request #39524 (panbingkun)
• links to: [Github] Pull Request #39524 (panbingkun)
• links to: [Github] Pull Request #39564 (huaxingao)
• links to: [Github] Pull Request #39564 (huaxingao)
• links to: [Github] Pull Request #39597 (huaxingao)