[HADOOP-18657] Tune ABFS create() retry logic

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.3.5
• Component/s: fs/azure
• Labels: pull-request-available
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 08/Mar/23 14:23
• Updated: 12/May/23 09:22

Description

Based on experience trying to debug this happening

1. add debug statements when create() fails
2. generated exception text to reference string shared with tests, path and error code
3. generated exception to include inner exception for full stack trace

Currently the retry logic is

1. create(overwrite=false)
2. if HTTP_CONFLICT/409 raised; call HEAD
3. use etag in create(path, overwrite=true, etag)
4. special handling of error HTTP_PRECON_FAILED = 412

There's a race condition here, which is if between 1 and 2 the file which exists is deleted. The retry should succeed, but currently a 404 from the head is escalated to a failure

proposed changes

1. if HEAD is 404, leave etag == null and continue
2. special handling of 412 also to handle 409

Issue Links

• links to: GitHub Pull Request #5462