Critical Description
In a stress test, I've seen the following counter-productive behavior:
- a leader is trying to send operations to a replica (eg a 10MB batch)
- the network is constrained due to other activity, so sending 10MB may take >1sec
- the request times out on the client side, likely while it was still in the process of sending the batch
- when the server receives it, it is likely to have timed out while waiting in the queue. Or ,it will receive it and upon processing will all be duplicate ops from the previous attempt
- the client has no idea whether the server received it or not, and thus keeps retrying the same batch (triggering the same timeout)
This tends to be a "sticky"/cascading sort of state: after one such timeout, the follower will be lagging behind more, and the next batch will be larger (up to the configured max batch size). The client neither backs off nor increases its timeout, so it will basically just keep the network pipe full of useless redundant updates
