Type: New Feature
Resolution: Won't Fix
Affects Version/s: None
Fix Version/s: None
Improve CouchDB replicator
- Allow running a large number of replication jobs
- Improve API with a focus on ease of use and performance. Avoid updating replication document with transient state updates. Instead create a proper API for querying replication states. At the same time provide a compatibility mode to let users keep existing behavior (of getting updates in documents).
- Improve network resource usage and performance. Multiple connection to the same cluster could share socket connections
- Handle rate limiting on target and source HTTP endpoints. Let replication request auto-discover rate limit capacity based on a proven algorithm such as Additive Increase / Multiplicative Decrease feedback control loop.
- Improve performance by avoiding repeatedly retrying failing replication jobs. Instead use exponential backoff.
- Improve recovery from long (but temporary) network failure. Currently if replications jobs fail to start 10 times in a row they will not be retried anymore. This is not always desirable. In case of a long enough DNS (or other network) failure replication jobs will effectively stop until they are manually restarted.
- Better handling of filtered replications: Failing to fetch filters could block couch replicator manager, lead to message queue backups and memory exhaustion. Also, when replication filter code changes update replication accordingly (replication job ID should change in that case)
- Provide better metrics to introspect replicator behavior.