[FLINK-9752] Add an S3 RecoverableWriter

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.7.0
• Component/s: Connectors / Common
• Labels: pull-request-available
• Assignee: Kostas Kloudas
• Reporter: Stephan Ewen
• Created: 04/Jul/18 20:14
• Updated: 18/Jan/19 07:41
• Resolved: 31/Oct/18 09:51

Description

S3 offers persistence only when uploads are complete. That means at the end of simple uploads and uploads of parts of a MultiPartUpload.

We should implement a RecoverableWriter for S3 that does a MultiPartUpload with a Part per checkpoint.
Recovering the reader needs the MultiPartUploadID and the list of ETags of previous parts.

We need additional staging of data in Flink state to work around the fact that

• Parts in a MultiPartUpload must be at least 5MB
• Part sizes must be known up front. (Note that data can still be streamed in the upload)

Issue Links

• is duplicated by: FLINK-6306 Sink for eventually consistent file systems
• is duplicated by: FLINK-8939 Provide better support for saving streaming data to s3
• is related to: FLINK-11388 Add an OSS RecoverableWriter
• links to: GitHub Pull Request #6795