[HADOOP-18792] s3a prefetching to use split start/end options to limit prefetch range

• Type: Sub-task
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 05/Jul/23 13:26
• Updated: 12/Jul/23 13:03

Description

the bundled hadoop record readers pass the split start/end down to openFile() -these can be used by the prefetching stream as hints as to where to start and stop prefetching.

Hints only as

• records can extend past split end; that last record will still be read when split end < EOF
• for formats with footers which get read, caching that footer is important