Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
4.0.0
Description
Currently, the SQL config spark.sql.files.ignoreMissingFiles influences on RDDs created in Spark SQL such as FileScanRDD but doesn't impact on HadoopRDD and NewHadoopRDD. The last RDDs have separate core config spark.files.ignoreMissingFiles. That inconsistency might confuse users if they don't know implementation details. This ticket aims to eliminate the inconsistency.
Attachments
Issue Links
- is related to
-
SPARK-45340 Remove the SQL config spark.sql.hive.verifyPartitionPath
- Resolved
- links to