[HADOOP-18805] s3a large file prefetch tests are too slow, don't validate data

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.9
• Fix Version/s: 3.3.9
• Component/s: fs/s3, test
• Labels: pull-request-available
• Assignee: Viraj Jasani
• Reporter: Steve Loughran
• Created: 14/Jul/23 17:47
• Updated: 26/Jul/23 14:01
• Resolved: 26/Jul/23 10:33

Description

the large file prefetch tests (including LRU cache eviction) are really slow.

moving under -scale may hide the problem for most runs, but they are still too slow, can time out, etc etc.

also, and this is very, very important, they can't validate the data.

Better:

• test on smaller files by setting a very small block size (1k bytes or less) just to force paged reads of a small 16k file.
• with known contents to the values of all forms of read can be validated
• maybe the LRU tests can work with a fake remote object which can then be used in a unit test
• extend one of the huge file tests to read from there -including s3-CSE encryption coverage.

Issue Links

• depends upon: HADOOP-18246 Remove lower limit on s3a prefetching/caching block size
• links to: GitHub Pull Request #5851
• links to: GitHub Pull Request #5887