Currently when we build a DNS routed service there is a parameter assigned called maximum IP address which determines how many cache IPs in each cache group should be handed out to the DNS system for the client. This determines the number of caches across which content is assigned. For small services we typically assign 1-2 caches for larger services we assign 3-5 caches.
This is currently done manually and if a service grows rapidly they could be handling a large volume of traffic and only have one cache in each group assigned.
I would like to consider some type of feedback loop which gages the popularity of a service and dynamically adjusts the number of cache IPs handed out.
We should also consider splitting up this behavior of tying DNS address hand out to the number of caches across which we hash content. These should be separate configuration and only the number of caches to hash across needs to be dynamic.