[HADOOP-17631] Configuration ${env.VAR:-FALLBACK} should eval FALLBACK when restrictSystemProps=true

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.4.0, 3.3.2
• Component/s: common
• Labels: pull-request-available
• Target Version/s: 3.4.0
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 09/Apr/21 13:36
• Updated: 06/Jul/22 10:42
• Resolved: 08/Jun/21 21:13

Description

When configuration reads in resources with a restricted parser, it skips evaluaging system ${env. } vars. But it also skips evaluating fallbacks

As a result, a property like fs.s3a.buffer.dir

${env.LOCAL_DIRS:-${hadoop.tmp.dir}} ends up evaluating as ${env.LOCAL_DIRS:-${hadoop.tmp.dir}}

It should instead fall back to the "env var unset" option of ${hadoop.tmp.dir}. This allows for configs (like for s3a buffer dirs) which are usable in restricted mode as well as unrestricted deployments.

Issue Links

• blocks: HADOOP-17386 fs.s3a.buffer.dir to be under Yarn container path on yarn applications
• is related to: HADOOP-18313 AliyunOSS: AliyunOSSBlockOutputStream should not mark the temporary file for deletion
• relates to: HADOOP-9642 Configuration to resolve environment variables via ${env.VARIABLE} references
• links to: GitHub Pull Request #2977