[SPARK-40039] Introducing a streaming checkpoint file manager based on Hadoop's Abortable interface

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: Structured Streaming
• Assignee: Attila Zsolt Piros
• Reporter: Attila Zsolt Piros
• Created: 11/Aug/22 00:39
• Updated: 12/Dec/22 18:10
• Resolved: 29/Aug/22 01:43

Description

Currently on S3 the checkpoint file manager (called FileContextBasedCheckpointFileManager) is based on rename. So when a file is opened for an atomic stream a temporary file used instead and when the stream is committed the file is renamed.

But on S3 a rename will be a file copy. So it has some serious performance implication.

But on Hadoop 3 there is new interface introduce called Abortable and S3AFileSystem has this capability which is implemented by on top S3's multipart upload. So when the file is committed a POST is sent (https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html) and when aborted a DELETE will be send
(https://docs.aws.amazon.com/AmazonS3/latest/API/API_AbortMultipartUpload.html)

Issue Links

• is depended upon by: SPARK-38445 Are hadoop committers used in Structured Streaming?
• links to: [Github] Pull Request #37474 (attilapiros)
• links to: [Github] Pull Request #37474 (attilapiros)
• links to: [Github] Pull Request #37677 (roczei)
• links to: [Github] Pull Request #37677 (roczei)
• links to: [Github] Pull Request #37687 (attilapiros)