[SPARK-48292] Revert [SPARK-39195][SQL] Spark OutputCommitCoordinator should abort stage when committed file not consistent with task status

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0, 3.5.2, 3.4.4
• Component/s: SQL
• Labels: pull-request-available
• Assignee: angerszhu
• Reporter: L. C. Hsieh
• Created: 15/May/24 18:15
• Updated: 04/Aug/24 19:12
• Resolved: 30/May/24 16:50

Description

When a task attemp fails but it is authorized to do task commit, OutputCommitCoordinator will make the stage failed with a reason message which says that task commit success, but actually the driver never knows if a task commit is successful or not. We should update the reason message to make it less confused.

See https://github.com/apache/spark/pull/36564#discussion_r1598660630

Issue Links

• relates to: SPARK-39195 Spark OutputCommitCoordinator should abort stage when committed file not consistent with task status
• links to: GitHub Pull Request #46696