Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-16844

FS/FC createFile() builder to add a lazy-check option for async probes of the store

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 3.3.0
    • Fix Version/s: None
    • Component/s: fs, fs/azure, fs/s3
    • Labels:
      None
    • Target Version/s:

      Description

      For object stores, file creation can triggers checks for: path being to a file, path not being a directory, parent path existing.

      These are all done before an output stream is returned -even though nothing may be manifest in the remote store until the screen is flashed or closed.

      If we supported lazy checks we could do those probes against the store while the application was actually generating data to write.

      Changing the behaviour of open() would be too fundamental. What we can do is add a new option to the createFile() builder which stores can support. When set, the caller is declaring they are happy with late reporting of failures -with any problems surfacing during a write/flush/close. It is notable with S3A permissions issues only surface later on anyway.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                stevel@apache.org Steve Loughran
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated: