[FLINK-25413] Use append dfs.nameservices hadoop config to replace override

• Type: Improvement
• Status: Open
• Priority: Not a Priority
• Resolution: Unresolved
• Labels: auto-deprioritized-major auto-deprioritized-minor pull-request-available
• Assignee: Unassigned
• Reporter: qiunan
• Created: 22/Dec/21 07:57
• Updated: 10/Aug/23 22:35

Description

In FLINK-16005[flink-yarn] Support yarn and hadoop config override.

In flink-conf.yaml

flink.hadoop.dfs.namenode.rpc-address.nameservice1.nn1: bigdata1:8020
flink.hadoop.dfs.namenode.rpc-address.nameservice1.nn2: bigdata2:8020
flink.hadoop.dfs.client.failover.proxy.provider.nameservice1: org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
flink.hadoop.dfs.ha.namenodes.nameservice1: nn1,nn2

flink.hadoop.dfs.nameservices: nameservice1

code：

// Approach 4: Flink configuration
// add all configuration key with prefix 'flink.hadoop.' in flink conf to hadoop conf
for (String key : flinkConfiguration.keySet()) {
    for (String prefix : FLINK_CONFIG_PREFIXES) {
        if (key.startsWith(prefix)) {
            String newKey = key.substring(prefix.length());
            String value = flinkConfiguration.getString(key, null);
            result.set(newKey, value);
            LOG.debug(
                    "Adding Flink config entry for {} as {}={} to Hadoop config",
                    key,
                    newKey,
                    value);
            foundHadoopConfiguration = true;
        }
    }
} 

If my HADOOP_CONF_DIR hdfs-site.xml have dfs.nameservices: nameservice2, see the code logic this config will be override. I think this config should not be override append will be better. if override we should add all config, but we have many clusters in production, it is impossible to configure all configurations in flink-conf.yaml.

 

Issue Links

• links to: GitHub Pull Request #18187