Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-16202

Enhance openFile() for better read performance against object stores

    XMLWordPrintableJSON

Details

    Description

      The openFile() builder API lets us add new options when reading a file

      Add an option "fs.s3a.open.option.length" which takes a long and allows the length of the file to be declared. If set, no check for the existence of the file is issued when opening the file

      Also: withFileStatus() to take any FileStatus implementation, rather than only S3AFileStatus -and not check that the path matches the path being opened. Needed to support viewFS-style wrapping and mounting.

      and Adopt where appropriate to stop clusters with S3A reads switched to random IO from killing download/localization

      • fs shell copyToLocal
      • distcp
      • IOUtils.copy

      Attachments

        Issue Links

          Activity

            People

              stevel@apache.org Steve Loughran
              stevel@apache.org Steve Loughran
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 23h 10m
                  23h 10m