[HADOOP-18528] Disable abfs prefetching by default

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.4
• Fix Version/s: 3.4.0, 3.3.5
• Component/s: fs/azure
• Labels: pull-request-available
• Target Version/s: 3.3.5
• Hadoop Flags: Reviewed

Release Note

ABFS block prefetching has been disabled to avoid HADOOP-18521 and buffer sharing on multithreaded processes (Hive, Spark etc). This will have little/no performance impact on queries against Parquet or ORC data, but can slow down sequential stream processing, including CSV files -however, the read data will be correct.
It may slow down distcp downloads, where the race condition does not arise. For maximum distcp performance re-enable the readahead by setting fs.abfs.enable.readahead to true.

• Assignee: Mehakmeet Singh
• Reporter: Mehakmeet Singh
• Created: 14/Nov/22 13:30
• Updated: 27/Jan/24 09:04
• Resolved: 15/Nov/22 14:38

Description

After the addition of HADOOP-18517, we should disable readAhead by default to mitigate inconsistent read results caused by ABFS prefetching, HADOOP-18521.

As an urgent fix: Disable readAhead/prefetch, tracked for 3.3.5.
Long-term fix: HADOOP-18521, tracked for 3.3.6.

Issue Links

• is caused by: HADOOP-17156 Clear abfs readahead requests on stream close
• relates to: HADOOP-18521 ABFS ReadBufferManager buffer sharing across concurrent HTTP requests
• links to: GitHub Pull Request #5134