Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-18568 Add Support for Azure Data Lake Store Gen 2 in File Sink
  3. FLINK-30588

Improvement in truncate code in the new ABFSOutputstream

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • None
    • None

    Description

      There are few TODOs in the new ABFSOutputstream. 

      1) Currently the buffer to read and do the truncate operation from the source file is 2MB. This we can allow some configuration based on the future usage and perf results.

      2) Currently while writing to temp file we write the entire data and then finally close/flush the data. Based on usages we can call intermediate flush/sync on the files. 

      Attachments

        Activity

          People

            Unassigned Unassigned
            ram_krish ramkrishna.s.vasudevan
            Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: