[SPARK-46761] quoted strings in a JSON path should support ? characters

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0, 4.0.0
• Fix Version/s: 4.0.0
• Component/s: SQL
• Labels: pull-request-available
• Assignee: Robert Joseph Evans
• Reporter: Robert Joseph Evans
• Created: 18/Jan/24 17:19
• Updated: 13/Mar/24 07:17
• Resolved: 13/Mar/24 07:17

Description

I think this impacts all versions of Spark after SPARK-18677, which made the operator work at all in 2.1.0/2.0.3

I comes down to

 name <- '.' ~> "[^\\.\\[]+".r | "['" ~> "[^\\'\\?]+".r <~ "']"

https://github.com/apache/spark/blob/01bb1b1a3dbfc68f41d9b13de863d26d587c7e2f/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala#L79

 

The regular expression/pattern is saying that we want a [' followed by one or more characters that are not a single quote ' or a question mark ? followed by ']. That question mark looks out of place. When I try to put in a question mark in a quoted string it fails to produce any result, but when I put the same data/path into https://jsonpath.com/ I get a result

 

data

{"?":"QUESTION"} 

path

$['?'] 

 

I also see no tests validating that a question mark is not allowed so I suspect that it is a long standing bug.

Issue Links

• links to: GitHub Pull Request #45420