[SPARK-50276] Add JSON path expression using the COLON operator

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 4.0.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Chenghao Lyu
• Created: 09/Nov/24 05:44
• Updated: 11/Nov/24 20:40

Description

Support JSON path expression by adding COLON sign operator [1] and JSON path expression [2] 

 [1] https://docs.databricks.com/en/sql/language-manual/functions/colonsign.html 

 [2] https://docs.databricks.com/en/sql/language-manual/sql-ref-json-path-expression.html

 

Syntax:

jsonExpr : jsonPath

jsonExpr can be either a VariantType or a valid JSON String.

• when jsonExpr is a valid JSON String, use an inner function semi_structured_extract_json_multi that is to be added with a new RegexParser such that the jsonPath is more flexible than  get_json_object).
• when jsonExpr is a VariantType, apply the built-in function variant_get