Major Description
This can cause replicas to get out of sync because replication may have failed, but then if you replay you can setup peer sync to pass when it should not.
After talking to Yonik a bit, I tried doing everything the same as on local fs except skipping the truncate - because of error handling we already have, this appears to work.
