[HADOOP-18246] Remove lower limit on s3a prefetching/caching block size

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0, 3.3.9
• Component/s: fs/s3
• Labels: pull-request-available
• Target Version/s: 3.4.0, 3.3.9
• Hadoop Flags: Reviewed
• Assignee: Ankit Saurabh
• Reporter: Daniel Carl Jones
• Created: 19/May/22 15:46
• Updated: 28/Jan/24 02:20
• Resolved: 02/Feb/23 18:45

Description

The minimum allowed block size currently is PREFETCH_BLOCK_DEFAULT_SIZE (8MB).

this.prefetchBlockSize = intOption(
                    conf, PREFETCH_BLOCK_SIZE_KEY, PREFETCH_BLOCK_DEFAULT_SIZE, PREFETCH_BLOCK_DEFAULT_SIZE);

https://github.com/apache/hadoop/blob/3aa03e0eb95bbcb066144706e06509f0e0549196/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L487-L488

Why is this the case and should we lower or remove it?

Issue Links

• is depended upon by: HADOOP-18805 s3a large file prefetch tests are too slow, don't validate data
• links to: GitHub Pull Request #4498
• links to: GitHub Pull Request #5120