[HDFS-6376] Distcp data between two HA clusters requires another configuration

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.2.0, 2.3.0, 2.4.0
• Fix Version/s: 2.6.0
• Component/s: datanode, federation, hdfs-client

Environment

Hadoop 2.3.0

• Target Version/s: 2.6.0
• Hadoop Flags: Reviewed

Release Note

Allow distcp to copy data between HA clusters. Users can use a new configuration property "dfs.internal.nameservices" to explicitly specify the name services belonging to the local cluster, while continue using the configuration property "dfs.nameservices" to specify all the name services in the local and remote clusters.

• Assignee: Dave Marion
• Reporter: Dave Marion
• Created: 12/May/14 21:56
• Updated: 24/May/16 20:14
• Resolved: 05/Sep/14 17:50

Description

User has to create a third set of configuration files for distcp when transferring data between two HA clusters.

Consider the scenario in [1]. You cannot put all of the required properties in core-site.xml and hdfs-site.xml for the client to resolve the location of both active namenodes. If you do, then the datanodes from cluster A may join cluster B. I can not find a configuration option that tells the datanodes to federate blocks for only one of the clusters in the configuration.

[1] http://mail-archives.apache.org/mod_mbox/hadoop-user/201404.mbox/%3CBAY172-W2133964E0C283968C161DD1520%40phx.gbl%3E

Issue Links

• breaks: HDFS-9365 Balancer does not work with the HDFS-6376 HA setup
• relates to: HDFS-9868 Add ability for DistCp to run between 2 clusters