[SPARK-1677] Allow users to avoid Hadoop output checks if desired

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 1.0.1, 1.1.0
• Component/s: Spark Core
• Target Version/s: 1.0.1, 1.1.0
• Assignee: Nan Zhu
• Reporter: Patrick Wendell
• Created: 30/Apr/14 05:42
• Updated: 23/Nov/16 01:14
• Resolved: 05/Jun/14 18:39

Description

For compatibility with older versions of Spark it would be nice to have an option `spark.hadoop.validateOutputSpecs` (default true) and a description "If set to true, validates the output specification used in saveAsHadoopFile and other variants. This can be disabled to silence exceptions due to pre-existing output directories."

This would just wrap the checking done in this PR:
https://issues.apache.org/jira/browse/SPARK-1100
https://github.com/apache/spark/pull/11

By first checking the spark conf.

Issue Links

• relates to: SPARK-1993 Let users skip checking output directory
• relates to: SPARK-2039 Run hadoop output checks for all formats
• links to: [Github] Pull Request #947 (CodingCat)