[HADOOP-18876] ABFS: Change default from disk to bytebuffer for fs.azure.data.blocks.buffer

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.6
• Fix Version/s: 3.4.0
• Component/s: build
• Labels: pull-request-available
• Target Version/s: 3.4.0
• Hadoop Flags: Reviewed

Release Note

The default value for fs.azure.data.blocks.buffer is changed from "disk" to "bytebuffer"

This will speed up writing to azure storage, at the risk of running out of memory
-especially if there are many threads writing to abfs at the same time and the
upload bandwidth is limited.

If jobs do run out of memory writing to abfs, change the option back to "disk"

• Assignee: Anmol Asrani
• Reporter: Anmol Asrani
• Created: 31/Aug/23 11:35
• Updated: 27/Jan/24 08:09
• Resolved: 09/Oct/23 15:52

Description

Change default from disk to bytebuffer for fs.azure.data.blocks.buffer.

Gathered from multiple workload runs, the presented data underscores a noteworthy enhancement in performance. The adoption of ByteBuffer for reading operations exhibited a remarkable improvement of approximately 64.83% when compared to traditional disk-based reading. Similarly, the implementation of ByteBuffer for write operations yielded a substantial efficiency gain of about 60.75%. These findings underscore the consistent and substantial advantages of integrating ByteBuffer across a range of workload scenarios.

Issue Links

• links to: GitHub Pull Request #6009