Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-18568 Add Support for Azure Data Lake Store Gen 2 in File Sink
  3. FLINK-30128

Introduce Azure Data Lake Gen2 APIs in the Hadoop Recoverable path

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Done
    • 1.13.1
    • 1.17.0
    • None
    • FileSink now supports Azure Data Lake Storage Gen2 APIs (`abfs://` and `abfss://`).

    Description

      Currently the HadoopRecoverableWriter assumes that the underlying FS is Hadoop and so it checks for DistributedFileSystem. It also tries to do a truncate and ensure the lease is recovered before the 'rename' operation is done.
      In the Azure Data lake gen 2 world, the driver does not support truncate and lease recovery API. We should be able to get the last committed size and if it matches go for the rename. Will be back with more details here.

      Attachments

        Issue Links

          causes

          Bug - A problem which impairs or prevents the functions of the product. FLINK-35531 Avoid calling hsync in flush method in BaseHadoopFsRecoverableFsDataOutputStream

          • Major - Major loss of function.
          • Closed
          links to

          Web Link GitHub Pull Request #21508

          Web Link GitHub Pull Request #21701

          Activity

            People

              ram_krish ramkrishna.s.vasudevan
              ram_krish ramkrishna.s.vasudevan
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: