[SPARK-33019] Use spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version=1 by default

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 3.0.0, 3.0.1, 3.1.0
• Fix Version/s: 3.0.2, 3.1.0
• Component/s: Spark Core
• Labels: correctness
• Target Version/s: 3.0.2, 3.1.0
• Assignee: Dongjoon Hyun
• Reporter: Dongjoon Hyun
• Created: 28/Sep/20 18:40
• Updated: 07/Oct/20 21:01
• Resolved: 29/Sep/20 19:03

Description

By default, Spark should use a safe file output committer algorithm to avoid MAPREDUCE-7282.

Issue Links

• relates to: MAPREDUCE-7282 MR v2 commit algorithm should be deprecated and not the default
• relates to: MAPREDUCE-7300 PathOutputCommitter to add method failedTaskAttemptCommitRecoverable()
• links to: [Github] Pull Request #29895 (dongjoon-hyun)