I think the fix for this is simple enough.
If a client calls getReplicaVisibleLength on a DN, and the DN replies with a ReplicaNotFoundException, then either:
a) the client just hasn't started writing the block yet,
or b) the client has stale block locations
In order to separate (a) from (b) I think we can iterate through the DNs, and see if all of the DNs have the same response. If they do, and the LocatedBlocks indicated a 0 length, then I think we can safely just act the same as if they returned length 0. Stale block locations are impossible (or at least very very unlikely) since we just called getBlockLocations from the NN.
My only question is whether we need to actually iterate through all of the DNs, or if we can just return 0 immediately on receiving ReplicaNotFoundException from the primary. I think going through all is actually important, because there may have been a concurrent pipeline recovery, in which case the old primary may have deleted the now-invalidated replica. (ie we can't distinguish between not having gotten the block yet and having gotten and already deleted the block)
Does this sound right? I will work on a patch if it does.