Apache Whirr (retired)
  1. Apache Whirr (retired)
  2. WHIRR-604

Non-resolvable hostnames should be reset to something resolvable

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.7.1
    • Fix Version/s: 0.8.0
    • Component/s: core
    • Labels:
      None

      Description

      So we currently have a hack in core/src/main/resources/functions/configure_hostnames.sh to reset the instance hostname on Rackspace to a-b-c-d.static.cloud-ips.com, since the hostname on the instance that Rackspace uses either isn't resolvable at all or isn't resolvable externally (not 100% sure which is the case there - I think they're just not resolvable). This logic should be generalized, since there are other providers where this is the case, and it's also an issue with, say, something like a private Cloudstack install.

      What seems to make sense is to add a property - something like whirr.ip-hostname-domain - and to add a check if we can actually resolve the instance's hostname from anywhere but that particular instance (since the very fact of it being in /etc/hostname means it'll be resolvable locally). If not, set the hostname to a-b-c-d.$

      {whirr.ip-hostname-domain}

      , with logic in Java somewhere to have a mapping of known provider->ip hostname domains for cases like Rackspace, so that whirr.ip-hostname-domain doesn't need to be set explicitly in those cases.

      1. WHIRR-604.patch
        10 kB
        Andrew Bayer
      2. WHIRR-604.patch
        10 kB
        Andrew Bayer
      3. WHIRR-604.patch
        10 kB
        Andrew Bayer
      4. WHIRR-604.patch
        10 kB
        Andrew Bayer
      5. WHIRR-604.patch
        10 kB
        Andrew Bayer
      6. WHIRR-604.patch
        18 kB
        Andrew Bayer

        Activity

        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open In Progress In Progress
        3d 27m 1 Adrian Cole (Inactive) 28/Jul/12 19:57
        In Progress In Progress Resolved Resolved
        17d 23h 47m 1 Andrew Bayer 15/Aug/12 19:45
        Hide
        Graham Gear added a comment -

        Tom, no problem, please see WHIRR-634 and WHIRR-635. Thanks, Graham

        Show
        Graham Gear added a comment - Tom, no problem, please see WHIRR-634 and WHIRR-635 . Thanks, Graham
        Hide
        Tom White added a comment -

        Graham, your additions sound really useful. Would you be able to open a new JIRA and attach your work there? Thanks.

        Show
        Tom White added a comment - Graham, your additions sound really useful. Would you be able to open a new JIRA and attach your work there? Thanks.
        Andrew Bayer made changes -
        Status In Progress [ 3 ] Resolved [ 5 ]
        Assignee Adrian Cole [ adrian@jclouds.org ] Andrew Bayer [ abayer ]
        Resolution Fixed [ 1 ]
        Hide
        Andrew Bayer added a comment -

        Committed.

        Show
        Andrew Bayer added a comment - Committed.
        Hide
        Tom White added a comment -

        +1

        Show
        Tom White added a comment - +1
        Andrew Bayer made changes -
        Attachment WHIRR-604.patch [ 12541106 ]
        Hide
        Andrew Bayer added a comment -

        Fixed.

        Show
        Andrew Bayer added a comment - Fixed.
        Hide
        Tom White added a comment -

        You have cloudservers-uk twice in setAutoHostnameForProvider. How about adding a unit test for it? Otherwise looks good.

        Show
        Tom White added a comment - You have cloudservers-uk twice in setAutoHostnameForProvider. How about adding a unit test for it? Otherwise looks good.
        Andrew Bayer made changes -
        Attachment WHIRR-604.patch [ 12541099 ]
        Hide
        Andrew Bayer added a comment -

        Final patch.

        Show
        Andrew Bayer added a comment - Final patch.
        Hide
        Andrew Bayer added a comment -

        I'll get the London thing in, but the rest would make sense as a separate JIRA, if that's ok. I'm also yanking the Rackspace next-gen special logic, as they're apparently not doing PTR records going forward. Sigh.

        Show
        Andrew Bayer added a comment - I'll get the London thing in, but the rest would make sense as a separate JIRA, if that's ok. I'm also yanking the Rackspace next-gen special logic, as they're apparently not doing PTR records going forward. Sigh.
        Hide
        Graham Gear added a comment -

        Great work, this addresses an issue I have been grappling with and keen to see it resolved.

        A couple of comments:

        • Existing configure_hostnames.sh works only for Debian derivatives, patch improves on this but should also update /etc/sysconfig/network with the hostname for RHEL derivatives
        • The Rackspace LON availability zone (cloudservers-uk) actually uses '.static.cloud-ips.co.uk' domain names not '.com', it would be good if this was built into the logic or paramaterised in some way
        • We should also update the private/local IP/host details in /etc/hosts, not just the public details to keep hostname in sync across interfaces

        I have a hacked configure_hostnames.sh meeting these needs today and can contribute if helpful.

        As an aside, Rackspace are moving toward an OpenStack implementation and a second generation cloud API/UI. Although aspects of this move are still in Beta (GA soon I believe) I have noticed that this comes with changes to the initial host/DNS configurations which will likely require us to revisit this problem in the future.

        Show
        Graham Gear added a comment - Great work, this addresses an issue I have been grappling with and keen to see it resolved. A couple of comments: Existing configure_hostnames.sh works only for Debian derivatives, patch improves on this but should also update /etc/sysconfig/network with the hostname for RHEL derivatives The Rackspace LON availability zone (cloudservers-uk) actually uses '.static.cloud-ips.co.uk' domain names not '.com', it would be good if this was built into the logic or paramaterised in some way We should also update the private/local IP/host details in /etc/hosts, not just the public details to keep hostname in sync across interfaces I have a hacked configure_hostnames.sh meeting these needs today and can contribute if helpful. As an aside, Rackspace are moving toward an OpenStack implementation and a second generation cloud API/UI. Although aspects of this move are still in Beta (GA soon I believe) I have noticed that this comes with changes to the initial host/DNS configurations which will likely require us to revisit this problem in the future.
        Andrew Bayer made changes -
        Attachment WHIRR-604.patch [ 12540907 ]
        Hide
        Andrew Bayer added a comment -

        assert fixed, and I dealt with a checkstyle error I missed.

        Show
        Andrew Bayer added a comment - assert fixed, and I dealt with a checkstyle error I missed.
        Hide
        Tom White added a comment -

        This is a definite improvement over the current code, even if it's not Adrian's more general approach. Adrian - are you OK with this approach for the time being?

        One small thing I noticed - the order of the arguments in the JUnit asserts is wrong - it should be assertEquals(expected, actual). Worth adding a message too.

        Show
        Tom White added a comment - This is a definite improvement over the current code, even if it's not Adrian's more general approach. Adrian - are you OK with this approach for the time being? One small thing I noticed - the order of the arguments in the JUnit asserts is wrong - it should be assertEquals(expected, actual) . Worth adding a message too.
        Andrew Bayer made changes -
        Attachment WHIRR-604.patch [ 12540755 ]
        Hide
        Andrew Bayer added a comment -

        Whoops - test failure that I should have noticed but missed because I added the test but hadn't actually built after adding the test. I'm dumb. That exposed an actual problem in the logic too, so go me. Fixed patch is here.

        Show
        Andrew Bayer added a comment - Whoops - test failure that I should have noticed but missed because I added the test but hadn't actually built after adding the test. I'm dumb. That exposed an actual problem in the logic too, so go me. Fixed patch is here.
        Andrew Bayer made changes -
        Attachment WHIRR-604.patch [ 12540752 ]
        Hide
        Andrew Bayer added a comment -

        there we go - no need for the beforeConfigure hackery now.

        Show
        Andrew Bayer added a comment - there we go - no need for the beforeConfigure hackery now.
        Hide
        Andrew Bayer added a comment -

        ...oh, wait, PRIVATE_IP was always actually the public IP. Guess what I can go change!

        Show
        Andrew Bayer added a comment - ...oh, wait, PRIVATE_IP was always actually the public IP. Guess what I can go change!
        Andrew Bayer made changes -
        Attachment WHIRR-604.patch [ 12540751 ]
        Hide
        Andrew Bayer added a comment -

        So this is a very admittedly hackish way to handle this, on some level, but here's a patch. I add two properties to ClusterSpec - whirr.auto-hostname-prefix and whirr.auto-hostname-suffix, along with some logic to set those automatically if the provider is one of a known list (cloudservers and rackspace-cloudservers currently). Then configure_hostnames.sh looks for AUTO_HOSTNAME_SUFFIX (and AUTO_HOSTNAME_PREFIX, but that's not set to anything but "" for cloudservers/rackspace-cloudservers - we need it here at Cloudera for our internal cloud, though) and if it's set, uses that to set the hostname.

        This meant I had to add configure_hostnames to a bunch of beforeConfigure methods (since it really should be using public IP, not private IP, which isn't set 'til we have the instance). That may be something we can work around, though. I also made a few other minor tweaks (like having HadoopProxy use an IP for SSH'ing, rather than a hostname, because...well, I've been bit by IPs not resolving often enough now that I want this). I've tested it with our internal cloud successfully and am running cloudservers-us tests shortly.

        Show
        Andrew Bayer added a comment - So this is a very admittedly hackish way to handle this, on some level, but here's a patch. I add two properties to ClusterSpec - whirr.auto-hostname-prefix and whirr.auto-hostname-suffix, along with some logic to set those automatically if the provider is one of a known list (cloudservers and rackspace-cloudservers currently). Then configure_hostnames.sh looks for AUTO_HOSTNAME_SUFFIX (and AUTO_HOSTNAME_PREFIX, but that's not set to anything but "" for cloudservers/rackspace-cloudservers - we need it here at Cloudera for our internal cloud, though) and if it's set, uses that to set the hostname. This meant I had to add configure_hostnames to a bunch of beforeConfigure methods (since it really should be using public IP, not private IP, which isn't set 'til we have the instance). That may be something we can work around, though. I also made a few other minor tweaks (like having HadoopProxy use an IP for SSH'ing, rather than a hostname, because...well, I've been bit by IPs not resolving often enough now that I want this). I've tested it with our internal cloud successfully and am running cloudservers-us tests shortly.
        Hide
        Adrian Cole (Inactive) added a comment -

        gist of WIP design: https://gist.github.com/3202856

        Show
        Adrian Cole (Inactive) added a comment - gist of WIP design: https://gist.github.com/3202856
        Hide
        Adrian Cole (Inactive) added a comment -

        actually, jclouds NodeMetadata.hostname is just that: the hostname of the machine (doesn't necessarily imply DNS)

        in clouds like EC2, the public (and private) dns names are supplied via api call

        http://docs.amazonwebservices.com/AWSEC2/latest/APIReference/ApiReference-ItemType-RunningInstancesItemType.html

        so..
        Instance.getPublicHostName actually attempts DNS resolution, so we can safely assume it is intended to be a DNS name
        Instance.getPrivateHostName is deprecated for some reason, so I presume we are to use for Instance.getPrivateIp in configuration needing LAN communication.

        If I'm correct(ish) so far, than the DnsResolver class could get us out of the problem where we don't know the correct value of the DNS name. Ex. failing DNS resolution, it could just fallback to x-x-x-x.static.cloud-ips.com, as there's nothing rackspace specific about that. Failing that (ex. in private clouds), or per override, publicHostname could be alternately specified.

        and again.. we'd end up setting this to the correct value on the machine via configure_hostnames.sh

        If we validate a general logic to apply, I think we can get rid of provider-specific logic completely.

        Summing up above:

        1. configure_hostnames.sh has the responsibility of adding instance.publicHostname, such that it resolves locally.
        2. the responsibility of setting instance.publicHostname to a valid value is Whirr. This logic follows the following

        • if an override is specified, set instance.publicHostname to that
        • if publicIp resolves via dns, return that.
        • if x-x-x-x.static.cloud-ips.com resolves via dns, return that
        • else return nodeMetadata.getHostname()
        Show
        Adrian Cole (Inactive) added a comment - actually, jclouds NodeMetadata.hostname is just that: the hostname of the machine (doesn't necessarily imply DNS) in clouds like EC2, the public (and private) dns names are supplied via api call http://docs.amazonwebservices.com/AWSEC2/latest/APIReference/ApiReference-ItemType-RunningInstancesItemType.html so.. Instance.getPublicHostName actually attempts DNS resolution, so we can safely assume it is intended to be a DNS name Instance.getPrivateHostName is deprecated for some reason, so I presume we are to use for Instance.getPrivateIp in configuration needing LAN communication. If I'm correct(ish) so far, than the DnsResolver class could get us out of the problem where we don't know the correct value of the DNS name. Ex. failing DNS resolution, it could just fallback to x-x-x-x.static.cloud-ips.com, as there's nothing rackspace specific about that. Failing that (ex. in private clouds), or per override, publicHostname could be alternately specified. and again.. we'd end up setting this to the correct value on the machine via configure_hostnames.sh If we validate a general logic to apply, I think we can get rid of provider-specific logic completely. Summing up above: 1. configure_hostnames.sh has the responsibility of adding instance.publicHostname, such that it resolves locally. 2. the responsibility of setting instance.publicHostname to a valid value is Whirr. This logic follows the following if an override is specified, set instance.publicHostname to that if publicIp resolves via dns, return that. if x-x-x-x.static.cloud-ips.com resolves via dns, return that else return nodeMetadata.getHostname()
        Hide
        Andrew Bayer added a comment -

        Yeah, I think so, on top of setting the hostname to something useful in the first place.

        Show
        Andrew Bayer added a comment - Yeah, I think so, on top of setting the hostname to something useful in the first place.
        Hide
        Adrian Cole (Inactive) added a comment -

        so to clarify the intent..

        basically, we need publicHostName -> publicIp and privateHostName -> privateIp regardless of whether the cloud is public or private.

        This is similar to ohai's
        :public_hostname -> :public_ipv4
        :local_hostname -> :local_ipv4

        https://github.com/opscode/ohai/blob/master/lib/ohai/plugins/rackspace.rb

        Right now, jclouds NodeMetadata has a hostname field, but (yet) not a publicHostname field

        Granted above, we want configure_hostnames.sh to ensure that the public (and probably also local) hostnames are in /etc/hosts.

        Do I have it right?

        Show
        Adrian Cole (Inactive) added a comment - so to clarify the intent.. basically, we need publicHostName -> publicIp and privateHostName -> privateIp regardless of whether the cloud is public or private. This is similar to ohai's :public_hostname -> :public_ipv4 :local_hostname -> :local_ipv4 https://github.com/opscode/ohai/blob/master/lib/ohai/plugins/rackspace.rb Right now, jclouds NodeMetadata has a hostname field, but (yet) not a publicHostname field Granted above, we want configure_hostnames.sh to ensure that the public (and probably also local) hostnames are in /etc/hosts. Do I have it right?
        Hide
        Andrew Bayer added a comment -

        ...that actually makes a lot more sense. I was wondering how the hell that'd work with the private IP.

        Show
        Andrew Bayer added a comment - ...that actually makes a lot more sense. I was wondering how the hell that'd work with the private IP.
        Hide
        Adrian Cole (Inactive) added a comment -

        first problem is that configure_hostnames.sh is really misleading. The variable PRIVATE_IP in configure_hostnames.sh is actually set with the public ip address!!

        Show
        Adrian Cole (Inactive) added a comment - first problem is that configure_hostnames.sh is really misleading. The variable PRIVATE_IP in configure_hostnames.sh is actually set with the public ip address!!
        Adrian Cole (Inactive) made changes -
        Field Original Value New Value
        Status Open [ 1 ] In Progress [ 3 ]
        Andrew Bayer created issue -

          People

          • Assignee:
            Andrew Bayer
            Reporter:
            Andrew Bayer
          • Votes:
            2 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development