[HADOOP-19098] Vector IO: consistent specified rejection of overlapping ranges

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.6
• Fix Version/s: 3.3.9, 3.5.0, 3.4.1
• Component/s: fs, fs/s3
• Labels: pull-request-available
• Target Version/s: 3.5.0
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 01/Mar/24 14:54
• Updated: 16/Aug/24 19:06
• Resolved: 10/Apr/24 12:43

Description

Related to PARQUET-2171 q: "how do you deal with overlapping ranges?"

I believe s3a rejects this, but the other impls may not.

Proposed

FS spec to say

• "overlap triggers IllegalArgumentException".
• special case: 0 byte ranges may be short circuited to return empty buffer even without checking file length etc.

Contract tests to validate this

(+ common helper code to do this).

I'll copy the validation stuff into the parquet PR for consistency with older releases

Issue Links

• causes: HADOOP-19204 VectorIO regression: empty ranges are now rejected
• causes: HADOOP-19253 Google GCS changes fail due to VectorIO changes
• contains: HADOOP-19101 Vectored Read into off-heap buffer broken in fallback implementation
• relates to: PARQUET-2171 Implement vectored IO in parquet file format
• links to: GitHub Pull Request #6604
• links to: GitHub Pull Request #6698
• links to: GitHub Pull Request #6700
• links to: GitHub Pull Request #6701
• links to: GitHub Pull Request #6706