[SPARK-41862] Fix a correctness bug in existence DEFAULT value lookups for the Orc data source

• Type: Sub-task
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: SQL
• Target Version/s: 3.4.0
• Assignee: Daniel
• Reporter: Daniel
• Created: 03/Jan/23 18:42
• Updated: 04/Jan/23 04:37
• Resolved: 04/Jan/23 00:30

Description

This issue aims to fix a correctness bug in existence DEFAULT value lookups during data source scans

Initially, SPARK-39862 tried to fix this, but was reverted via SPARK-41858 due to the perf issue.

Issue Links

• relates to: SPARK-39862 Fix bug in existence DEFAULT value lookups for V2 data sources
• relates to: SPARK-41858 Fix ORC reader perf regression due to DEFAULT value feature
• links to: [Github] Pull Request #39370 (dtenedor)
• links to: [Github] Pull Request #39370 (dtenedor)
• links to: [Github] Pull Request #39380 (dongjoon-hyun)
• links to: [Github] Pull Request #39380 (dongjoon-hyun)