Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15620 Über-jira: S3A phase VI: Hadoop 3.3 features
  3. HADOOP-16759

Filesystem openFile() builder to take a FileStatus param

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 3.3.0
    • Fix Version/s: 3.3.0
    • Component/s: fs, fs/azure, fs/s3
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Let us pass in a file status to openFile() so that S3A & ABFS will skip their own HEAD requests just to see if a file is there, a normal file and get its length + etag,

          CompletableFuture<FSDataInputStream> streamF = fs.openFile(stat.getPath())
            .withFileStatus(stat).build()
      

      code opening files off a listing of everything in a directory can eliminate a lot of requests here.

      Also: change the specification of openFile's completable future to say "returned stream may only raise FNFE, access restrictions on the first read"

      That is: it's not just potentially an async open, it's possibly lazy evaluated entirely.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                stevel@apache.org Steve Loughran
                Reporter:
                stevel@apache.org Steve Loughran
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: