Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-6376

Distcp data between two HA clusters requires another configuration

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.2.0, 2.3.0, 2.4.0
    • Fix Version/s: 2.6.0
    • Labels:
      None
    • Environment:

      Hadoop 2.3.0

    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Allow distcp to copy data between HA clusters. Users can use a new configuration property "dfs.internal.nameservices" to explicitly specify the name services belonging to the local cluster, while continue using the configuration property "dfs.nameservices" to specify all the name services in the local and remote clusters.
      Show
      Allow distcp to copy data between HA clusters. Users can use a new configuration property "dfs.internal.nameservices" to explicitly specify the name services belonging to the local cluster, while continue using the configuration property "dfs.nameservices" to specify all the name services in the local and remote clusters.

      Description

      User has to create a third set of configuration files for distcp when transferring data between two HA clusters.

      Consider the scenario in [1]. You cannot put all of the required properties in core-site.xml and hdfs-site.xml for the client to resolve the location of both active namenodes. If you do, then the datanodes from cluster A may join cluster B. I can not find a configuration option that tells the datanodes to federate blocks for only one of the clusters in the configuration.

      [1] http://mail-archives.apache.org/mod_mbox/hadoop-user/201404.mbox/%3CBAY172-W2133964E0C283968C161DD1520%40phx.gbl%3E

        Attachments

        1. HDFS-6376.000.patch
          16 kB
          Haohui Mai
        2. HDFS-6376.008.patch
          14 kB
          Jing Zhao
        3. HDFS-6376.009.patch
          16 kB
          Haohui Mai
        4. HDFS-6376.010.patch
          17 kB
          Haohui Mai
        5. HDFS-6376.011.patch
          17 kB
          Haohui Mai
        6. HDFS-6376-2.patch
          8 kB
          Dave Marion
        7. HDFS-6376-3-branch-2.4.patch
          8 kB
          Dave Marion
        8. HDFS-6376-4-branch-2.4.patch
          8 kB
          Dave Marion
        9. HDFS-6376-5-trunk.patch
          6 kB
          Dave Marion
        10. HDFS-6376-6-trunk.patch
          10 kB
          Dave Marion
        11. HDFS-6376-7-trunk.patch
          6 kB
          Dave Marion
        12. HDFS-6376-branch-2.4.patch
          8 kB
          Dave Marion
        13. HDFS-6376-patch-1.patch
          8 kB
          Dave Marion

          Issue Links

            Activity

              People

              • Assignee:
                dlmarion Dave Marion
                Reporter:
                dlmarion Dave Marion
              • Votes:
                1 Vote for this issue
                Watchers:
                17 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: