[YARN-10410] Does not work specifying the docker client configuration in distributed shell

• Type: Bug
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.2
• Component/s: applications/distributed-shell
• Assignee: Unassigned
• Reporter: KWON BYUNGCHANG
• Created: 26/Aug/20 09:34
• Updated: 26/Aug/20 12:56

Description

Job sumbit client side log  

2020-08-26 12:07:30,756 INFO distributedshell.Client: Got dt for hdfs://at; Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:at, Ident: (token for magnum: HDFS_DELEGATION_TOKEN owner=[e-mail], renewer=yarn, realUser=, issueDate=1598411249950, maxDate=1599275249950, sequenceNumber=1359401, masterKeyId=461)
2020-08-26 12:07:30,757 INFO distributedshell.Client: Got dt for hdfs://at; Kind: kms-dt, Service: kms://[e-mail]:9393/kms, Ident: (kms-dt owner=magnum, renewer=yarn, realUser=, issueDate=1598411250569, maxDate=1599016050569, sequenceNumber=2791602, masterKeyId=908)
2020-08-26 12:07:30,969 INFO util.DockerClientConfigHandler: Token read from Docker client configuration file: Kind: DOCKER_CLIENT_CREDENTIAL_TOKEN, Service: reg.test.com, Ident: (registryUrl: "reg.test.com" applicationId: "application_1593772175687_475926")
 

 

AM LOG

20/08/26 12:07:33 INFO distributedshell.ApplicationMaster: Executing with tokens:
20/08/26 12:07:33 INFO distributedshell.ApplicationMaster: Kind: YARN_AM_RM_TOKEN, Service: , Ident: (appAttemptId { application_id { id: 475926 cluster_timestamp: 1593772175687 } attemptId: 1 } keyId: 1589057158)
20/08/26 12:07:33 INFO distributedshell.ApplicationMaster: Kind: kms-dt, Service: kms://[e-mail]:9393/kms, Ident: (kms-dt owner=magnum, renewer=yarn, realUser=, issueDate=1598411250569, maxDate=1599016050569, sequenceNumber=2791602, masterKeyId=908)
20/08/26 12:07:33 INFO distributedshell.ApplicationMaster: Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:at, Ident: (token for magnum: HDFS_DELEGATION_TOKEN owner=[e-mail], renewer=yarn, realUser=, issueDate=1598411249950, maxDate=1599275249950, sequenceNumber=1359401, masterKeyId=461)
20/08/26 12:07:33 INFO impl.NMClientAsyncImpl: Upper bound of the thread pool size is 500
20/08/26 12:07:33 WARN distributedshell.ApplicationMaster: Timeline service is not enabled

 

On the client side in distributea shell, Client read docker_client_config and put it in credentials instances
DOCKER_CLIENT_CREDENTIAL_TOKEN is not propagated to the container.
eventually docker container launch that requires auth token failed. 

 

Issue Links

• is related to: YARN-5428 Allow for specifying the docker client configuration directory