[SPARK-37750] ANSI mode: optionally return null result if element not exists in array/map

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.3.0
• Component/s: SQL
• Epic Link: ANSI enhancements in Spark 3.3
• Assignee: Gengliang Wang
• Reporter: Gengliang Wang
• Created: 27/Dec/21 08:17
• Updated: 11/Apr/22 12:49
• Resolved: 04/Jan/22 11:24

Description

Add a new configuration `spark.sql.ansi.failOnElementNotExists` which controls whether throwing exceptions or returning null results when element not exists in the [] operator in array/map type.

The default value of the new configuration is true.

Issue Links

• links to: [Github] Pull Request #35031 (gengliangwang)
• links to: [Github] Pull Request #35031 (gengliangwang)
• links to: [Github] Pull Request #35099 (tedyu)

Sub-Tasks

• Turn spark.sql.ansi.strictIndexOperator into internal config: Resolved