[HADOOP-19093] Improve rate limiting through ABFS in Manifest Committer

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.4.0
• Component/s: fs/azure, test
• Labels: pull-request-available
• Target Version/s: 3.5.0
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 27/Feb/24 12:49
• Updated: 06/Jun/24 19:38

Description

I need a load test to verify that the rename resilience of the manifest committer actually works as intended

• test suite with name ILoadTest* prefix (as with s3)
• parallel test running with many threads doing many renames
• verify that rename recovery should be detected
• and that all renames MUST NOT fail.

maybe also: metrics for this in fs and doc update.
Possibly; LogExactlyOnce to warn of load issues

Issue Links

• is related to: MAPREDUCE-7474 [ABFS] Improve commit resilience and performance in Manifest Committer
• links to: GitHub Pull Request #6596