Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-7544

part of the data is transferred over eth0 even after configuring eth1 in mapred-site.xml and hdfs-site.xml

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 0.20.2, 0.20.203.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:

      Description

      I configure dfs.datanode.dns.interface & mapred.tasktracker.dns.interface to use eth1.
      I then run terasort and used dstat to monitor the traffic on each of my interfaces.
      Aparently, about 50% of the data is still being transferred over eth0.

      The problem is only "solved" in case I manually change the hostnames and add appropriate entries in /etc/hosts so resolving host's name - in my cluster - reaches eth1. However, changing the hostname is inappropriate for my production environment.

      NOTE: This issue might be rlated to https://issues.apache.org/jira/browse/HADOOP-6797 . However, we know that our network is propely configured, and still it doesn't guarantee that reverseDns will succeed. Hence, we suffer a lot from this problem.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              shani ml Shani Afrimi
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: