[SPARK-23308] ignoreCorruptFiles should not ignore retryable IOException

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Won't Fix
• Affects Version/s: 2.2.1
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Márcio Furlani Carmona
• Created: 01/Feb/18 22:21
• Updated: 14/Aug/18 03:23
• Resolved: 14/Aug/18 03:23

Description

When `spark.sql.files.ignoreCorruptFiles` is set it totally ignores any kind of RuntimeException or IOException, but some possible IOExceptions may happen even if the file is not corrupted.

One example is the SocketTimeoutException which can be retried to possibly fetch the data without meaning the data is corrupted.

 

See: 

https://github.com/apache/spark/blob/e30e2698a2193f0bbdcd4edb884710819ab6397c/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileScanRDD.scala#L163

Issue Links

• relates to: HADOOP-15216 S3AInputStream to handle reconnect on read() failure better