[YARN-7677] Docker image cannot set HADOOP_CONF_DIR

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.1.0, 2.10.1
• Labels: Docker
• Hadoop Flags: Incompatible change

Release Note

The HADOOP_CONF_DIR environment variable is no longer unconditionally inherited by containers even if it does not appear in the nodemanager whitelist variables specified by the yarn.nodemanager.env-whitelist property. If the whitelist property has been modified from the default to not include HADOOP_CONF_DIR yet containers need it to be inherited from the nodemanager's environment then the whitelist settings need to be updated to include HADOOP_CONF_DIR.

• Assignee: Jim Brennan
• Reporter: Eric Badger
• Created: 20/Dec/17 19:15
• Updated: 06/Aug/20 19:53
• Resolved: 06/Aug/20 19:42

Description

Currently, HADOOP_CONF_DIR is being put into the task environment whether it's set by the user or not. It completely bypasses the whitelist and so there is no way for a task to not have HADOOP_CONF_DIR set. This causes problems in the Docker use case where Docker containers will set up their own environment and have their own HADOOP_CONF_DIR preset in the image itself.

Issue Links

• breaks: MAPREDUCE-7055 MR jobs are failing with Could not find or load main class for MRAppMaster
• is blocked by: YARN-5714 ContainerExecutor does not order environment map
• relates to: YARN-3611 Support Docker Containers In LinuxContainerExecutor
• relates to: OOZIE-3492 [spark-action] Missing HADOOP_CONF_DIR property