Major Description
As of now (May 2018), the kudu tablet change_config move_replica tool or the rebalancer tool (https://gerrit.cloudera.org/#/c/10399/) both have special provision to take care of a source leader replica: they make stepping down after the non-voter replica has caught up. That leaves the space for situations with over-replicated tablets when the tool is interrupted after it set the 'REPLACE' attribute but before the leader source replica is gone.
Making the catalog manager taking care of the leader replicas marked with the 'REPLACE' attribute would help to clean up tablets' configurations in such cases.