[HADOOP-13203] S3A: Support fadvise "random" mode for high performance readPositioned() reads

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.8.0
• Fix Version/s: 2.8.0, 3.0.0-alpha1
• Component/s: fs/s3
• Target Version/s: 2.8.0
• Hadoop Flags: Reviewed

Release Note

S3A has added support for configurable input policies. Similar to fadvise, this configuration provides applications with a way to specify their expected access pattern (sequential or random) while reading a file. S3A then performs optimizations tailored to that access pattern. See site documentation of the fs.s3a.experimental.input.fadvise configuration property for more details. Please be advised that this feature is experimental and subject to backward-incompatible changes in future releases.

• Assignee: Rajesh Balamohan
• Reporter: Rajesh Balamohan
• Created: 25/May/16 07:33
• Updated: 22/Oct/20 18:51
• Resolved: 22/Jun/16 15:02

Description

Currently file's "contentLength" is set as the "requestedStreamLen", when invoking S3AInputStream::reopen(). As a part of lazySeek(), sometimes the stream had to be closed and reopened. But lots of times the stream was closed with abort() causing the internal http connection to be unusable. This incurs lots of connection establishment cost in some jobs. It would be good to set the correct value for the stream length to avoid connection aborts.

I will post the patch once aws tests passes in my machine.

Issue Links

• incorporates: HADOOP-13286 add a S3A scale test to do gunzip and linecount
• is related to: HADOOP-16241 S3AInputStream PositionReadable should perform ranged read on dedicated stream
• is related to: HDFS-2744 Extend FSDataInputStream to allow fadvise
• relates to: HADOOP-13028 add low level counter metrics for S3A; use in read performance tests
• relates to: HADOOP-14965 s3a input stream "normal" fadvise mode to be adaptive