[SPARK-41448] Make consistent MR job IDs in FileBatchWriter and FileFormatWriter

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0, 3.1.2, 3.2.2, 3.3.1
• Fix Version/s: 3.2.3, 3.3.2, 3.4.0
• Component/s: Spark Core, SQL
• Assignee: Hui An
• Reporter: Hui An
• Created: 08/Dec/22 10:22
• Updated: 18/Feb/23 14:48
• Resolved: 12/Dec/22 10:25

Description

SPARK-26873 fix the consistent issue for FileFormatWriter, but SPARK-33402 break this requirement by introducing a random long, we need to address this to expects identical task IDs across attempts for correctness.

Also FileBatchWriter doesn't follow this requirement, need to fix it as well.

Issue Links

• links to: [Github] Pull Request #38980 (boneanxs)
• links to: GitHub Pull Request #4359