[SPARK-45316] Respect `spark.sql.files.ignoreMissingFiles` in HadoopRDD

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: SQL
• Labels: pull-request-available
• Assignee: Max Gekk
• Reporter: Max Gekk
• Created: 25/Sep/23 13:08
• Updated: 26/Sep/23 14:33
• Resolved: 26/Sep/23 14:33

Description

Currently, the SQL config spark.sql.files.ignoreMissingFiles influences on RDDs created in Spark SQL such as FileScanRDD but doesn't impact on HadoopRDD and NewHadoopRDD. The last RDDs have separate core config spark.files.ignoreMissingFiles. That inconsistency might confuse users if they don't know implementation details. This ticket aims to eliminate the inconsistency.

Issue Links

• is related to: SPARK-45340 Remove the SQL config spark.sql.hive.verifyPartitionPath
• links to: GitHub Pull Request #43097