[SPARK-38605] Retrying on file manager operation in HDFSMetadataLog

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 3.4.0
• Component/s: Structured Streaming
• Assignee: Unassigned
• Reporter: L. C. Hsieh
• Created: 19/Mar/22 20:13
• Updated: 30/Mar/22 04:49
• Resolved: 30/Mar/22 04:49

Description

Currently HDFSMetadataLog uses CheckpointFileManager to do some file operation like opening metadata file. It is very easy to be affected by network blips and causes the streaming query failed. Although we can restart the streaming query, but it takes more time to recover.

Such file operations should be resilient with such situation by retrying.

Issue Links

• links to: [Github] Pull Request #35994 (viirya)