[YARN-11255] Support loading alternative docker client config from system environment

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: yarn
• Labels: pull-request-available
• Target Version/s: 3.4.0
• Hadoop Flags: Reviewed
• Assignee: Ashutosh Gupta
• Reporter: Ashutosh Gupta
• Created: 18/Aug/22 20:57
• Updated: 28/Jan/24 05:07
• Resolved: 21/Sep/22 11:24

Description

When using YARN docker support, although the hadoop shell supported 

-docker_client_config

 to pass the client config file that contains security token to generate the docker config for each job as a temporary file.

For other applications that submit jobs to YARN, e.g. Spark, which loads the docker setting via system environment e.g. 

spark.executorEnv.* 

will not be able to add those authorization token because this system environment isn't considered in YARN.

Add genetic solution to handle these kind of cases without making changes in spark code or others

Eg

When using remote container registry, the YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG must reference the config.json
file containing the credentials used to authenticate.

DOCKER_IMAGE_NAME=hadoop-docker 
DOCKER_CLIENT_CONFIG=hdfs:///user/hadoop/config.json
spark-submit --master yarn \
--deploy-mode cluster \
--conf spark.executorEnv.YARN_CONTAINER_RUNTIME_TYPE=docker \
--conf spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=$DOCKER_IMAGE_NAME \
--conf spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG=$DOCKER_CLIENT_CONFIG \
--conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_TYPE=docker \
--conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=$DOCKER_IMAGE_NAME \
--conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG=$DOCKER_CLIENT_CONFIG \
sparkR.R

Issue Links

• links to: GitHub Pull Request #4884