[HADOOP-16241] S3AInputStream PositionReadable should perform ranged read on dedicated stream

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: fs/s3
• Target Version/s: 3.5.0
• Assignee: Sahil Takiar
• Reporter: Sahil Takiar
• Created: 08/Apr/19 20:46
• Updated: 04/Jan/24 12:47

Description

The current implementation of PositionReadable in S3AInputStream is pretty close to the default implementation in FsInputStream.

This JIRA proposes overriding the read(long position, byte[] buffer, int offset, int length) method and re-implementing the readFully(long position, byte[] buffer, int offset, int length) method in S3A.

The new implementation would perform a "ranged read" on a dedicated object stream (rather than the shared one). Prototypes have shown this to bring a considerable performance improvement to readers who are only interested in reading a random chunk of the file at a time (e.g. Impala, although I would assume HBase would benefit from this as well).

Setting fs.s3a.experimental.input.fadvise to RANDOM is helpful for clients that rely on pread, but has a few drawbacks:

• Unless the client explicitly sets fadvise to RANDOM, they will get at least one connection reset when the backwards seek is issued (after which fadvise automatically switches to RANDOM)
• Data is only read in 64 kb chunks, so for a large read, several GET requests must be issued to S3 to fetch the data; while the 64 kb chunk value is configurable, it is hard to set a reasonable value for variable length preads
• If the readahead value is too big, closing the input stream can take considerable time because the stream has to be drained of data before it can be closed

The new implementation of PositionReadable would issue a GetObjectRequest with the range specified by position and the size of the given buffer. The data would be read from the S3ObjectInputStream and then closed at the end of the method. This stream would be independent of the wrappedStream currently maintained by S3A.

This brings the following benefits:

• The PositionedReadable methods can be thread-safe without a synchronized block, which allows clients to concurrently call pread methods on the same S3AInputStream instance
• preads will request all the data at once rather than requesting it in chunks via the readahead logic
• Avoids performing potentially expensive seeks when performing preads

Issue Links

• relates to: HADOOP-11867 Add a high-performance vectored read API.
• relates to: HADOOP-12444 Support lazy seek in S3AInputStream
• relates to: HADOOP-13028 add low level counter metrics for S3A; use in read performance tests
• relates to: HADOOP-13203 S3A: Support fadvise "random" mode for high performance readPositioned() reads