[HADOOP-17386] fs.s3a.buffer.dir to be under Yarn container path on yarn applications

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.3.0
• Component/s: fs/s3
• Assignee: Unassigned
• Reporter: Steve Loughran
• Created: 18/Nov/20 16:55
• Updated: 09/Apr/21 13:37

Description

1. fs.s3a.buffer.dir defaults to hadoop.tmp.dir which is /tmp or similar
2. we use this for storing file blocks during upload
3. staging committers use it for all files in a task, which can be a lot more
4. a lot of systems don't clean up /tmp until reboot -and if they stay up for a long time then they accrue files written through s3a staging committer from spark containers which fail

Fix: use ${env.LOCAL_DIRS:-${hadoop.tmp.dir}}/s3a as the option so that if env.LOCAL_DIRS is set is used over hadoop.tmp.dir. YARN-deployed apps will use that for the buffer dir. When the app container is destroyed, so is the directory.

Issue Links

• is blocked by: HADOOP-17631 Configuration ${env.VAR:-FALLBACK} should eval FALLBACK when restrictSystemProps=true