[YARN-9746] RM should merge local config for token renewal

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: Junfan Zhang
• Created: 14/Aug/19 03:54
• Updated: 15/Aug/19 12:09

Description

This issue links to YARN-5910.

When to support multi-cluster delegation token renew, the path of YARN-5910 works in most scenarios.

But when intergrating with Oozie, we encounter some problems. In Oozie having multi delegation tokens including HDFS_DELEGATION_TOKEN(another cluster HA token) and MR_DELEGATION_TOKEN(Oozie mr launcher token), to support renew another cluster's token, YARN-5910 was patched and related config was set. The config is as follows

<property>
                    <name>mapreduce.job.send-token-conf</name>
                    <value>dfs.namenode.kerberos.principal|dfs.nameservices|^dfs.namenode.rpc-address.*$|^dfs.ha.namenodes.*$|^dfs.client.failover.proxy.provider.*$</value>
                </property>
                <property>
                    <name>dfs.nameservices</name>
                    <value>hadoop-clusterA-ns01,hadoop-clusterA-ns02,hadoop-clusterA-ns03,hadoop-clusterA-ns04,hadoop-clusterB-ns01,hadoop-clusterB-ns02,hadoop-clusterB-ns03,hadoop-clusterB-ns04</value>
                </property>
                <property>
                    <name>dfs.ha.namenodes.hadoop-clusterB-ns01</name>
                    <value>nn1,nn2</value>
                </property>
                <property>
                    <name>dfs.namenode.rpc-address.hadoop-clusterB-ns01.nn1</name>
                    <value>namenode01-clusterB.hadoop:8020</value>
                </property>
                <property>
                    <name>dfs.namenode.rpc-address.hadoop-clusterB-ns01.nn2</name>
                    <value>namenode02-clusterB.hadoop:8020</value>
                </property>
                <property>
                    <name>dfs.client.failover.proxy.provider.hadoop-clusterB-ns01</name>
                    <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
                </property>

However, the MR_DELEGATION_TOKEN could‘t be renewed, because of lacking some config. Although we can set the required configurations through the app, this is not a good idea. So i think rm should only rewrite the jobConf passed by app to solve the above situation.  

Issue Links

• Blocked: YARN-5910 Support for multi-cluster delegation tokens