Given a write pipeline:
DN1 -> DN2 -> DN3
DN3 detected cheksum error and terminate, DN2 truncates its replica to the ACKed size. Then a new pipeline is attempted as
DN1 -> DN2 -> DN4
DN4 detects checksum error again. Later when replaced DN4 with DN5 (and so on), it failed for the same reason. This led to the observation that DN2's data is corrupted.
Found that the software currently truncates DN2's replca to the ACKed size after DN3 terminates. But it doesn't check the correctness of the data already written to disk.
So intuitively, a solution would be, when downstream DN (DN3 here) found checksum error, propagate this info back to upstream DN (DN2 here), DN2 checks the correctness of the data already written to disk, and truncate the replica to to MIN(correctDataSize, ACKedSize).
if the tail node in the pipeline detects a checksum error, then it returns a special error code back up the pipeline indicating this (rather than just disconnecting)
if a non-tail node receives this error code, then it immediately scans its own block on disk (from the beginning up through the last acked length). If it detects a corruption on its local copy, then it should assume that it is the faulty one, rather than the downstream neighbor. If it detects no corruption, then the faulty node is either the downstream mirror or the network link between the two, and the current behavior is reasonable.