[SPARK-29649] Stop task set if FileAlreadyExistsException was thrown when writing to output file

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: Spark Core, SQL
• Assignee: L. C. Hsieh
• Reporter: L. C. Hsieh
• Created: 30/Oct/19 06:50
• Updated: 14/Nov/19 02:01
• Resolved: 14/Nov/19 02:01

Description

We already know task attempts that do not clean up output files in staging directory can cause job failure (SPARK-27194). There was proposals trying to fix it by changing output filename, or deleting existing output files. These proposals are not reliable completely.

The difficulty is, as previous failed task attempt wrote the output file, at next task attempt the output file is still under same staging directory, even the output file name is different.

If the job will go to fail eventually, there is no point to re-run the task until max attempts are reached. For the jobs running a lot of time, re-running the task can waste a lot of time.

Issue Links

• links to: GitHub Pull Request #26312