[FLINK-11838] Create RecoverableWriter for GCS

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Done
• Affects Version/s: 1.8.0
• Fix Version/s: 1.15.0
• Component/s: Connectors / FileSystem
• Labels: pull-request-available usability
• Assignee: Galen Warren
• Reporter: Fokko Driesprong
• Created: 06/Mar/19 12:45
• Updated: 11/Nov/22 15:28
• Resolved: 20/Jan/22 01:44

Description

GCS supports the resumable upload which we can use to create a Recoverable writer similar to the S3 implementation:
https://cloud.google.com/storage/docs/json_api/v1/how-tos/resumable-upload

After using the Hadoop compatible interface: https://github.com/apache/flink/pull/7519
We've noticed that the current implementation relies heavily on the renaming of the files on the commit:
https://github.com/apache/flink/blob/master/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopRecoverableFsDataOutputStream.java#L233-L259
This is suboptimal on an object store such as GCS. Therefore we would like to implement a more GCS native RecoverableWriter

Issue Links

• causes: FLINK-25772 GCS filesystem fails license checker
• is related to: FLINK-25577 Update GCS documentation
• relates to: FLINK-19481 Add support for a flink native GCS FileSystem
• supercedes: FLINK-11378 Allow HadoopRecoverableWriter to write to Hadoop compatible Filesystems
• links to: GitHub Pull Request #7915
• links to: GitHub Pull Request #14875
• links to: GitHub Pull Request #15599
• links to: GitHub Pull Request #18275