[HADOOP-17166] ABFS: configure output stream thread pool

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.3.1
• Component/s: fs/azure
• Labels: pull-request-available
• Assignee: Bilahari T H
• Reporter: Bilahari T H
• Created: 30/Jul/20 12:46
• Updated: 05/Oct/21 16:04
• Resolved: 14/Oct/20 22:56

Description

fs.azure.write.max.concurrent.requests -number of concurrent requests a single stream may POST at
fs.azure.write.max.requests.to.queue -number of blocks which will be queued before the client blocks.

the amount of memory consumed is fs.azure.write.max.concurrent.requests * fs.azure.write.max.requests.to.queue (default= 8MB)

Issue Links

• is related to: HADOOP-17195 Intermittent OutOfMemory error while performing hdfs CopyFromLocal to abfs
• links to: GitHub Pull Request #2179
• links to: GitHub Pull Request #2209
• links to: PR Link