Right. My only point was that the scope of that work was limited.
ADDROLE is, indeed, right here. Forgot about that.
There is some infrastructure in place to force a leader onto the replica that has the preferredLeader property set, though it's not exposed at the collections API level. Perhaps add a parameter to the REBALANCELEADERS like shards=(comma separated list of shards) that would only force preferredLeader on the shards specified? That should map pretty easily into the current code. Whether we expose that to the end users or not is an open question, although I don't see any reason not to.
I think I like the AVOID_RESPONSIBILITY idea (DECOMISSIONING is another possibility maybe?) rather than specific roles, but that's not a strong preference. And you're doing the work so you get to decide .
Here's an idea from left field, just force the node into the "down" state and keep it there while doing the bookkeeping, including the deletereplica stuff? You'd have to insure that there was a live replica for each shard for each collection hosted on the node, but if so I think much of the other stuff is essentially automatic although I admit I don't know the details at this point. You'd also have to do the ADDREPLICA (step <4>) before shutting it down.... You could force the preferredLeader to another node for the relevant replicas, and if we add a shard list to REBALANCELEADERS force the leaders off the node before forcing it into the down state to control churn too.
<5> is unnecessary for any shard that has no replica with preferredLeader set on the node.
It's probably sufficient to just ADDREPLICAPROP to some other node or BALANCESHARDUNIQUE for preferredLeader and let it go at that. REBALANCELEADERS can be a fairly heavy-weight process, and the preferredLeader thing is just a suggestion anyway. If things are temporarily unbalanced in terms of actual leadership that's probably fine. Short of someone issuing REBALANCELEADERS, I really don't expect all the preferredLeader to always be the leader anyway. What happens is that when a replica comes up, it inserts itself at the beginning of the queue for being elected leader and only really gets to be leader when the current leader goes down or REBALANCELEADERS is issued for that collection. At least that's how I remember it....
Up to you of course. In fact it's not absolutely necessary to have any preferred leader either