[HADOOP-18313] AliyunOSS: AliyunOSSBlockOutputStream should not mark the temporary file for deletion

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.10.2, 3.2.4, 3.3.3
• Component/s: fs/oss
• Labels: pull-request-available
• Assignee: wujinhu
• Reporter: wujinhu
• Created: 25/Jun/22 10:09
• Updated: 04/Aug/22 04:04
• Resolved: 04/Aug/22 04:04

Description

AliyunOSSBlockOutputStream buffers data in local directory before uploading to OSS. It uses LocalDirAllocator.createTmpFileForWrite which will create a temp file on this directory and invoke DeleteOnExitHook.add to add the path to a set and will be deleted on VM exit through a shutdown hook. However, the size of set will be growing if the VM does not exit.

Issue Links

• relates to: HADOOP-17386 fs.s3a.buffer.dir to be under Yarn container path on yarn applications
• relates to: HADOOP-17631 Configuration ${env.VAR:-FALLBACK} should eval FALLBACK when restrictSystemProps=true
• links to: GitHub Pull Request #4502