[SPARK-24589] OutputCommitCoordinator may allow duplicate commits

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.2.1, 2.3.1
• Fix Version/s: 2.1.3, 2.2.2, 2.3.2, 2.4.0
• Component/s: Spark Core
• Target Version/s: 2.2.2, 2.3.2
• Assignee: Marcelo Masiero Vanzin
• Reporter: Marcelo Masiero Vanzin
• Created: 19/Jun/18 01:31
• Updated: 22/Jun/18 21:08
• Resolved: 21/Jun/18 21:17

Description

This is a sibling bug to SPARK-24552. While investigating the source of that bug, it was found that currently the output committer allows duplicate commits when there are stage retries, and the task with the task attempt number (one in each stage that currently has running tasks) try to commit their output.

This can lead to duplicate data in the output.

Issue Links

• links to: [Github] Pull Request #21577 (vanzin)
• links to: [Github] Pull Request #21607 (vanzin)