[YARN-8981] Virtual IP address support

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.0.2, 3.1.1
• Labels: DNS Docker docker service service-engine service-orchestration virtual_hosts
• Assignee: Unassigned
• Reporter: Ruslan Dautkhanov
• Created: 06/Nov/18 18:53
• Updated: 14/Nov/18 22:33

Description

I couldn't find support for virtual IP addresses in YARN framework. 

This would be great if we have a docker-on-yarn service and if it for example has to be failed over to another physical host, clients can still find it. 

So the idea is for YARN to bring up that virtual IP address (an additional/secondary IP address ) on a physical host where that particular docker container is running, so the clients that use that container's services don't have to change connection details every time that container moves around in YARN cluster.

Similarly to virtual IP addresses in Kubernetes world:

https://kubernetes.io/docs/concepts/services-networking/service/#virtual-ips-and-service-proxies

One implementation could be through `ip address add` \ `ip address remove`.

Kubernetes uses a more complicated `kube-proxy`, similarly to `docker-proxy` process in pure dockers / non-kubernetes docker deployments.

Another approach is running a separate DNS service for a DNS subdomain (main DNS server would have to forward all requests for that DNS subdomain to a YARN DNS service). In Oracle Clusterware similar process is called GNS:

https://docs.oracle.com/en/database/oracle/oracle-database/12.2/cwsol/about-the-grid-naming-service-vip-address.html#GUID-A4EE0CC6-A5F1-4507-82D6-D5C43E0F1584

Would be great to have support for either virtual IP addresses managed by YARN directly or something similar to Oracle's GNS dns service.