[SPARK-48286] Analyze 'exists' default expression instead of 'current' default expression in structField to v2 column conversion

• Type: Bug
• Status: Resolved
• Priority: Trivial
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0, 3.5.2
• Component/s: Spark Core
• Labels: pull-request-available
• Assignee: Uros Stankovic
• Reporter: Uros Stankovic
• Created: 15/May/24 09:13
• Updated: 17/Jun/24 18:38
• Resolved: 06/Jun/24 20:10

Description

org.apache.spark.sql.catalyst.util.ResolveDefaultColumns#analyze method accepts 3 parameter

1) Field to analyze
2) Statement type - String

3) Metadata key - CURRENT_DEFAULT or EXISTS_DEFAULT

Method org.apache.spark.sql.connector.catalog.CatalogV2Util#structFieldToV2Column
pass fieldToAnalyze and EXISTS_DEFAULT as second parameter, so it is not metadata key, instead of that, it is statement type, so bad expression is analyzed.

Issue Links

• links to: GitHub Pull Request #46594
• links to: GitHub Pull Request #46978