[HADOOP-17386] fs.s3a.buffer.dir to be under Yarn container path on yarn applications

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.3.9, 3.3.6
• Component/s: fs/s3
• Labels: pull-request-available
• Assignee: Monthon Klongklaew
• Reporter: Steve Loughran
• Created: 18/Nov/20 16:55
• Updated: 21/Jan/24 00:51
• Resolved: 22/Feb/22 04:51

Description

1. fs.s3a.buffer.dir defaults to hadoop.tmp.dir which is /tmp or similar
2. we use this for storing file blocks during upload
3. staging committers use it for all files in a task, which can be a lot more
4. a lot of systems don't clean up /tmp until reboot -and if they stay up for a long time then they accrue files written through s3a staging committer from spark containers which fail

Fix: use ${env.LOCAL_DIRS:-${hadoop.tmp.dir}}/s3a as the option so that if env.LOCAL_DIRS is set is used over hadoop.tmp.dir. YARN-deployed apps will use that for the buffer dir. When the app container is destroyed, so is the directory.

Issue Links

• is blocked by: HADOOP-17631 Configuration ${env.VAR:-FALLBACK} should eval FALLBACK when restrictSystemProps=true
• is related to: HADOOP-18313 AliyunOSS: AliyunOSSBlockOutputStream should not mark the temporary file for deletion
• is related to: HADOOP-18764 fs.azure.buffer.dir to be under Yarn container path on yarn applications
• links to: GitHub Pull Request #3908