Currently storage_port must be configured identically on all nodes in a cluster and it is assumed that this is the case when connecting to a remote node.
This prevents running in any environment that requires multiple nodes to be able to bind to the same network interface, such as with many automatic provisioning/deployment frameworks.
The current solutions seems to be
- use a separate network interface for each node deployed to the same box. This puts a big requirement on IP allocation at large scale.
- allow multiple clusters to be provisioned from the same resource pool, but restrict allocation to a maximum of one node per host from each cluster, assuming each cluster is running on a different storage port.
It would make operations much simpler in these kind of environments if the environment provisioning the resources could assign the ports to be used when bringing up a new node on shared hardware.
The changes required would be at least the following:
1. configure seeds as IP:port instead of just IP
2. gossip the storage port as part of a node's ApplicationState
3. refer internally to nodes by hostID instead of IP, since there will be multiple nodes with the same IP
(1) & (2) are mostly trivial and I already have a patch for these. The bulk of the work to enable this is (3), and I would structure this as a separate pre-requisite patch.