[HADOOP-18028] High performance S3A input stream with prefetching & caching

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Fix Version/s: 3.3.9
• Component/s: fs/s3
• Labels: pull-request-available

Language

java

• Assignee: Bhalchandra Pandit
• Reporter: Bhalchandra Pandit
• Created: 29/Nov/21 16:12
• Updated: 16/Jan/24 08:37

Description

I work for Pinterest. I developed a technique for vastly improving read throughput when reading from the S3 file system. It not only helps the sequential read case (like reading a SequenceFile) but also significantly improves read throughput of a random access case (like reading Parquet). This technique has been very useful in significantly improving efficiency of the data processing jobs at Pinterest. 
 
I would like to contribute that feature to Apache Hadoop. More details on this technique are available in this blog I wrote recently:
https://medium.com/pinterest-engineering/improving-efficiency-and-reducing-runtime-using-s3-read-optimization-b31da4b60fa0
 

Issue Links

• is depended upon by: HADOOP-18179 Boost S3A Stream Read Performance
• is depended upon by: HADOOP-18477 Über-jira: S3A Hadoop 3.3.9 features
• is related to: HIVE-25827 Parquet file footer is read multiple times, when multiple splits are created in same file
• links to: GitHub Pull Request #3736
• links to: GitHub Pull Request #4109
• links to: GitHub Pull Request #4654
• links to: GitHub Pull Request #4675
• links to: GitHub Pull Request #4752
• links to: GitHub Pull Request #5559
• links to: GitHub Pull Request #5605

Sub-Tasks

• test failures with prefetching s3a input stream: Resolved
• s3a prefetching stream to move off twitter FuturePool: Resolved
• document use and architecture design of prefetching s3a input stream: Resolved
• Remove use of scala jar twitter util-core with java futures in S3A prefetching stream: Resolved
• move org.apache.hadoop.fs.common package into hadoop-common module: Resolved
• S3File to store reference to active S3Object in a field.: Resolved
• s3a prefetching stream to support unbuffer(): In Progress
• tune logging of prefetch problems: Open
• s3a prefetching to use SemaphoredDelegatingExecutor for submitting work: Resolved
• Convert s3a prefetching to use JavaDoc for fields and enums: Resolved
• S3PrefetchingInputStream to support status probes when closed: Resolved
• Collect IOStatistics during S3A prefetching: Resolved
• Ensure S3A prefetching stream memory consumption scales: Open
• stream warns Not all bytes were read from the S3ObjectInputStream when closed: Resolved
• Use async drain threshold to decide b/w async and sync draining: Resolved
• tests in ITestS3AInputStreamPerformance are failing: Resolved
• Rebase s3a prefetching feature branch on top of trunk: Resolved
• Remove lower limit on s3a prefetching/caching block size: Resolved
• Tests in ITestS3AOpenCost are failing: Resolved
• Add in configuration option to enable prefetching: Resolved
• Review s3a prefetching input stream retry code; synchronization: Open
• S3A prefetch - Implement LRU cache for SingleFilePerBlockCache: Resolved
• Update class names to be clear they belong to S3A prefetching: Resolved
• S3A prefetching: Error logging during reads: Resolved
• hadoop-aws maven build to add a prefetch profile to run all tests with prefetching: Resolved
• Implement readFully(long position, byte[] buffer, int offset, int length): Resolved
• rebase feature/HADOOP-18028-s3a-prefetch to trunk: Resolved
• fs.s3a.prefetch.block.size to be read through longBytesOption: Resolved
• ITestS3AFileSystemStatistic failure in prefetch feature branch: Open
• ITestS3ACannedACLs failure; not in a span: Resolved
• S3A Prefetch - SingleFilePerBlockCache to use LocalDirAllocator: Resolved
• s3a prefetching Executor should be closed: Resolved
• assertion failure in ITestS3APrefetchingInputStream: Resolved
• Fix transient failure of ITestS3APrefetchingInputStream#testRandomReadLargeFile: Resolved
• Backport S3A prefetching stream to branch-3.3: Resolved
• s3a prefetch cache blocks should be accessed by RW locks: Resolved
• CachingBlockManager to use AtomicBoolean for closed flag: Resolved
• S3A prefetching: switch to prefetching for chosen read policies: Open
• s3a prefetching to use split start/end options to limit prefetch range: In Progress
• s3a large file prefetch tests are too slow, don't validate data: Resolved
• s3a prefetch read/write file operations should guard channel close: Resolved
• s3a prefetch LRU cache eviction metric: Resolved
• S3ACachingInputStream.ensureCurrentBuffer(): lazy seek means all reads look like random IO: Open
• ITestS3APrefetchingCacheFiles teardown failure if setup() fails: Open
• Use builder for prefetch CachingBlockManager: Resolved
• S3A prefetching to support Vector IO: Open
• TestS3ACachingBlockManager fails intermittently in Yetus: Open