[HADOOP-16879] s3a mkdirs() to not check dest for a dir marker

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 3.2.1
• Component/s: fs/s3
• Assignee: Unassigned
• Reporter: Steve Loughran
• Created: 24/Feb/20 17:48
• Updated: 03/Dec/20 11:45
• Resolved: 03/Dec/20 11:45

Description

S3A innerMkdirs() calls getFileStatus() to probe dest path for being a file or dir, then goes to reject/no-op.

The HEAD path + / in that code may add a 404 to the S3 load balancers, so subsequent probes for the path fail.

Proposed: only look for file then LIST underneath

if no entry found: probe for parent being a dir (LIST; HEAD + /), if true create the marker entry. If not, start the walk (or should we then check?)

This increases the cost of mkdir on an existing empty dir marker; reduces it on a non-empty dir. Creates dir markers above dir markers to avoid those cached 404s.

Issue Links

• duplicates: HADOOP-16804 s3a mkdir path/ can add 404 to S3 load balancers