We have observed that write fails permanently if the first packet doesn't go through properly and pipeline recovery happens. If the write op creates a pipeline, but the actual data packet does not reach one or more datanodes in time, the pipeline recovery will be done against the 0-byte partial block.
If additional datanodes are added, the block is transferred to the new nodes. After the transfer, each node will have a meta file containing the header and 0-length data block file. The pipeline recovery seems to work correctly up to this point, but write fails when actual data packet is resent.