Whirr
  1. Whirr
  2. WHIRR-492

Hadoop TaskTrackers on EC2 mix of internal and external host names

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.7.0
    • Fix Version/s: None
    • Component/s: service/hadoop
    • Labels:
      None
    • Environment:

      EC2

      Description

      Running a 1 master, 3 node cluster of Hadoop + HBase, I see on the Hadoop task trackers
      the following hosts:

      1) domU-12-31-39-0B-C1-81.compute-1.internal
      2) ip-10-66-197-215.ec2.internal
      3) domU-12-31-39-0E-F2-41.compute-1.internal

      This is consistent with both the trunk (today) and 0.7.0.2, and all
      tasks on node 2) above then fail.

      My config is the following:
      whirr.cluster-name=hbase
      whirr.instance-templates=1
      zookeeper+hadoop-namenode+hadoop-jobtracker+hbase-master,3
      hadoop-datanode+hadoop-tasktracker+hbase-regionserver
      whirr.provider=aws-ec2
      whirr.identity=$

      {env:AWS_ACCESS_KEY_ID}

      whirr.credential=$

      {env:AWS_SECRET_ACCESS_KEY}

      whirr.hardware-id=c1.xlarge
      whirr.image-id=us-east-1/ami-da0cf8b3
      whirr.location-id=us-east-1

        Activity

        Hide
        Andrei Savu added a comment -

        Any ideas on how we should incorporate this? Only as documentation?

        Show
        Andrei Savu added a comment - Any ideas on how we should incorporate this? Only as documentation?
        Hide
        Tim Robertson added a comment -

        Confirmed it appears to work consistently when using us-east-1b
        Thanks Tom, Andrei

        Show
        Tim Robertson added a comment - Confirmed it appears to work consistently when using us-east-1b Thanks Tom, Andrei
        Hide
        Andrei Savu added a comment -

        Good point! Tim can you give it a try using us-east-1a (or similar) as the location-id?

        Show
        Andrei Savu added a comment - Good point! Tim can you give it a try using us-east-1a (or similar) as the location-id?
        Hide
        Tom White added a comment -

        I think these are all internal addresses, just different DNS names. I wonder if it would help to constrain everything to one zone rather than a whole region: http://whirr.apache.org/faq.html#How_can_I_start_a_cluster_in_a_particular_location

        Show
        Tom White added a comment - I think these are all internal addresses, just different DNS names. I wonder if it would help to constrain everything to one zone rather than a whole region: http://whirr.apache.org/faq.html#How_can_I_start_a_cluster_in_a_particular_location
        Tim Robertson created issue -

          People

          • Assignee:
            Unassigned
            Reporter:
            Tim Robertson
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:

              Development