We observed a case where a replica's on disk length is less than acknowledged length, breaking the assumption in recovery code.
It turns out that if an exception is thrown within BlockReceiver#receivePacket, the in-memory replica on disk length may not be updated, but the data is written to disk anyway.
For example, here's one exception we observed
There are potentially other places and causes where an exception is thrown within BlockReceiver#receivePacket, so it may not make much sense to alleviate it for this particular exception. Instead, we should improve replica recovery code to handle the case where ondisk size is less than acknowledged size, and update in-memory checksum accordingly.