Hadoop Common
  1. Hadoop Common
  2. HADOOP-6210

Bind Hadoop RPC outgoing to a specific nic

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Won't Fix
    • Affects Version/s: 0.20.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      In environments with multiple interfaces on a machine, it would be useful to be able to bind outgoing traffic to specific interfaces rather than letting the outbound be picked at random. This is especially important for things like Solaris IPMP.

        Issue Links

          Activity

          Hide
          Matthew Byng-Maddick added a comment -

          Better than trying to bind it to a "single nic" you actually want to be able to specify the IP address - though in the case of v6 link-local and similar addresses - you also need to cope with scoping.

          Show
          Matthew Byng-Maddick added a comment - Better than trying to bind it to a "single nic" you actually want to be able to specify the IP address - though in the case of v6 link-local and similar addresses - you also need to cope with scoping.
          Hide
          Allen Wittenauer added a comment -

          True. I had in my head binding to a nic alias, actually.

          Show
          Allen Wittenauer added a comment - True. I had in my head binding to a nic alias, actually.
          Hide
          Matthew Byng-Maddick added a comment -

          Other bits within hadoop try that, but that's not always useful - it turns out that on Linux there are two ways to bind aliases to interfaces. You can either create an explicit alias interface - ie. one with a :n, such as eth0:1, or you can use the "iproute" bits to do: /sbin/ip addr add ..., which adds that IP as an alias on that interface - without creating a separate one. In typical linux style, this is only visible through the iproute interface (normally via /sbin/ip), and doesn't surface in ifconfig(8).

          The reason for needing to specify the IP address comes out in the VIP case - where you run an HA of some kind (in my case to co-locate the namenode/tasktracker with the slaves, and have the namenode always appear on the same IP address), then the entire point here is to stop the source address being the VIP - but the VIP is an alias on the interface you would normally communicate on. The whole thing ends up being a bit of a mess, unfortunately, and of course the whole thing about nic aliases is particularly platform specific - solaris uses the names of the drivers: hme0, e1000g0, nge0, FreeBSD similarly: fxp0 - and again on FreeBSD you can have alias IP addresses - I think solaris tends to prefer that you use the :1 explicit alias interface. Linux, of course, is described up above - having the best and worst of both worlds. All fun.

          Show
          Matthew Byng-Maddick added a comment - Other bits within hadoop try that, but that's not always useful - it turns out that on Linux there are two ways to bind aliases to interfaces. You can either create an explicit alias interface - ie. one with a : n , such as eth0:1, or you can use the "iproute" bits to do: /sbin/ip addr add ..., which adds that IP as an alias on that interface - without creating a separate one. In typical linux style, this is only visible through the iproute interface (normally via /sbin/ip), and doesn't surface in ifconfig(8). The reason for needing to specify the IP address comes out in the VIP case - where you run an HA of some kind (in my case to co-locate the namenode/tasktracker with the slaves, and have the namenode always appear on the same IP address), then the entire point here is to stop the source address being the VIP - but the VIP is an alias on the interface you would normally communicate on. The whole thing ends up being a bit of a mess, unfortunately, and of course the whole thing about nic aliases is particularly platform specific - solaris uses the names of the drivers: hme0, e1000g0, nge0, FreeBSD similarly: fxp0 - and again on FreeBSD you can have alias IP addresses - I think solaris tends to prefer that you use the :1 explicit alias interface. Linux, of course, is described up above - having the best and worst of both worlds. All fun.
          Hide
          Allen Wittenauer added a comment -

          No, no, it is actually better to bind to a specific IP address.

          I just mention that I was thinking of nic aliases when I wrote that because most of the non-hacky HA solutions I'm aware of use real, functional aliases to perform the ip movement from machine to machine.

          Show
          Allen Wittenauer added a comment - No, no, it is actually better to bind to a specific IP address. I just mention that I was thinking of nic aliases when I wrote that because most of the non-hacky HA solutions I'm aware of use real, functional aliases to perform the ip movement from machine to machine.
          Hide
          Matthew Byng-Maddick added a comment -

          When you say "real functional aliases" what do you mean - do you mean foon:m? If so, then that doesn't really work on, say, FreeBSD, where the correct way IS to bind addresses using ifconfig and the "alias" modifier - it doesn't have a concept of foon:m - there's a foon interface with a list of inet addresses bound and a list of inet6 addresses and their corresponding forwarder entries. Linux's 2 methods are both equally supported by the kernel, but less so by the operating system. It doesn't make one more or less right than the other, and actually, though it pains me to admit it, using iproute programmatically is significantly neater than searching to find a free binding on a host with many interface aliases.

          As I say, if you want to have, for example, a v6-link-local scoped address - you probably want to bind by scope (which is almost the same as binding by NIC but not quite).

          In v6 I suspect that this kind of problem will get worse, as it's reasonable to have link-local or site-local addresses bound along with real routeable addresses to interfaces. Even when dealing with v4 hadoop seems to generally choose the v4-in-v6 space (::ffff:xxx.xxx.xxx.xxx) for its bindings (so AF_INET6 rather than AF_INET) - on my linux installation, anyway.

          What I'm saying, really, is that you kind of need both, but probably prefer IP addresses.

          Show
          Matthew Byng-Maddick added a comment - When you say "real functional aliases" what do you mean - do you mean foon : m ? If so, then that doesn't really work on, say, FreeBSD, where the correct way IS to bind addresses using ifconfig and the "alias" modifier - it doesn't have a concept of foon : m - there's a foon interface with a list of inet addresses bound and a list of inet6 addresses and their corresponding forwarder entries. Linux's 2 methods are both equally supported by the kernel, but less so by the operating system. It doesn't make one more or less right than the other, and actually, though it pains me to admit it, using iproute programmatically is significantly neater than searching to find a free binding on a host with many interface aliases. As I say, if you want to have, for example, a v6-link-local scoped address - you probably want to bind by scope (which is almost the same as binding by NIC but not quite). In v6 I suspect that this kind of problem will get worse, as it's reasonable to have link-local or site-local addresses bound along with real routeable addresses to interfaces. Even when dealing with v4 hadoop seems to generally choose the v4-in-v6 space (::ffff:xxx.xxx.xxx.xxx) for its bindings (so AF_INET6 rather than AF_INET) - on my linux installation, anyway. What I'm saying, really, is that you kind of need both, but probably prefer IP addresses.

            People

            • Assignee:
              Unassigned
              Reporter:
              Allen Wittenauer
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development