[FLINK-30128] Introduce Azure Data Lake Gen2 APIs in the Hadoop Recoverable path

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Done
• Affects Version/s: 1.13.1
• Fix Version/s: 1.17.0
• Labels: pull-request-available
• Release Note: FileSink now supports Azure Data Lake Storage Gen2 APIs (`abfs://` and `abfss://`).
• Assignee: ramkrishna.s.vasudevan
• Reporter: ramkrishna.s.vasudevan
• Created: 22/Nov/22 04:42
• Updated: 5 days ago 14:23
• Resolved: 10/Jan/23 02:16

Description

Currently the HadoopRecoverableWriter assumes that the underlying FS is Hadoop and so it checks for DistributedFileSystem. It also tries to do a truncate and ensure the lease is recovered before the 'rename' operation is done.
In the Azure Data lake gen 2 world, the driver does not support truncate and lease recovery API. We should be able to get the last committed size and if it matches go for the rename. Will be back with more details here.

Issue Links

• causes: FLINK-35531 Avoid calling hsync in flush method in BaseHadoopFsRecoverableFsDataOutputStream
• links to: GitHub Pull Request #21508
• links to: GitHub Pull Request #21701