[CASSANDRA-15823] Support for networking via identity instead of IP

• Type: Improvement
• Status: Triage Needed
• Priority: Normal
• Resolution: Unresolved
• Labels: kubernetes
• Platform: All
• Assignee: Unassigned
• Reporter: Christopher Bradford
• Created: 19/May/20 23:28
• Updated: 24/Feb/21 17:01

Description

TL;DR: Instead of mapping host ids to IPs, use hostnames. This allows resolution to different IP addresses per DC that may then be forwarded to nodes on remote networks without requiring node to node IP connectivity for cross-dc links.

 

This approach should not affect existing deployments as those could continue to use IPs as the hostname and skip resolution.

---

With orchestration platforms like Kubernetes and the usage of ephemeral containers in environments today we should consider some changes to how we handle the tracking of nodes and their network location. Currently we maintain a mapping between host ids and IP addresses.

 

With traditional infrastructure, if a node goes down it, usually, comes back up with the same IP. In some environments this contract may be explicit with virtual IPs that may move between hosts. In newer deployments, like on Kubernetes, this contract is not possible. Pods (analogous to nodes) are assigned an IP address at start time. Should the pod be restarted or scheduled on a different host there is no guarantee we would have the same IP. Cassandra is protected here as we already have logic in place to update peers when we come up with the same host id, but a different IP address.

 

There are ways to get Kubernetes to assign a specific IP per Pod. Most recommendations involve the use of a service per pod. Communication with the fixed service IP would automatically forward to the associated pod, regardless of address. We could use this approach, but it seems like this would needlessly create a number of extra resources in our k8s cluster to get around the problem. Which, to be fair, doesn't seem like much of a problem with the aforementioned mitigations built into C*.

 

So what is the actual problem? Cross-region, cross-cloud, hybrid-deployment connectivity between pods is a pain. This can be solved with significant investment by those who want to deploy these types of topologies. You can definitely configure connectivity between clouds over dedicated connections, or VPN tunnels. With a big chunk of time insuring that pod to pod connectivity just works even if those pods are managed by separate control planes, but that again requires time and talent. There are a number of edge cases to support between the ever so slight, but very important, differences in cloud vendor networks.

 

Recently there have been a number of innovations that aid in the deployment and operation of these types of applications on Kubernetes. Service meshes support distributed microservices running across multiple k8s cluster control planes in disparate networks. Instead of directly connecting to IP addresses of remote services instead they use a hostname. With this approach, hostname traffic may then be routed to a proxy that sends traffic over the WAN (sometimes with mTLS) to another proxy pod in the remote cluster which then forwards the data along to the correct pod in that network. (See attached diagrams)

 

Which brings us to the point of this ticket. Instead of mapping host ids to IPs, use hostnames (and update the underlying address periodically instead of caching indefinitely). This allows resolution to different IP addresses per DC (k8s cluster) that may then be forwarded to nodes (pods) on remote networks (k8s clusters) without requiring node to node (pod to pod) IP connectivity between them. Traditional deployments can still function like they do today (even if operators opt to keep using IPs as identifiers instead of hostnames). This proxy approach is then enabled like those we see in service meshes.

 

Notes

C* already has the concept of broadcast addresses vs those which are bound on the node. This approach could be leveraged to provide the behavior we're looking for, but then the broadcast values would need to be pre-computed and match across all k8s control planes. By using hostnames the underlying IP address does not matter and will most likely be different in each cluster.

 

I recognize the title may be a bit misleading as we would obviously still communicate over TCP/IP., but it concisely conveys the point.

Issue Links

• is related to: CASSANDRA-4120 Gossip identifies hosts by UUID
• is related to: CASSANDRA-1518 remember ring state between restarts
• relates to: CASSANDRA-11559 Enhance node representation