[SPARK-40826] Add additional checkpoint rename file check

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: Structured Streaming
• Assignee: L. C. Hsieh
• Reporter: L. C. Hsieh
• Created: 18/Oct/22 03:14
• Updated: 19/Oct/22 05:06
• Resolved: 19/Oct/22 05:06

Description

We encountered an issue recently that one customer's structured streaming job failed to read delta file.

The temporary file exists but it was not successfully renamed to final delta file path.

We currently don't check if renamed file exists but assume it successful. As the result, failing to read delta file assumed to be committed in last batch makes re-triggering the job impossible.

We should be able to do a check against checkpoint renamed file to prevent such difficulty in advance.

Issue Links

• links to: [Github] Pull Request #38291 (viirya)
• links to: [Github] Pull Request #38291 (viirya)