It turns out there's a really big bug in pipeline construction for append... I'm not sure why more tests aren't failing more often!
Currently in BlockReceiver's constructor, we copy the reference to the provided block, and increment its generation stamp to the newGs. But, this is just a reference to the same Block object used in DataXceiver.opWriteBlock. So, when the DN tries to set up the append pipeline to its downstream mirror, it ends up passing the new generation stamp version of the block, and the downstream mirror will fail to construct a pipeline.
The end result is that we can never successfully construct a pipeline for append with more than one datanode in it! This test would fail about 2/3 of the time since it would see the old replica on one of the nodes that didn't make it into the append pipeline.
This patch fixees the BlockReceiver constructor to take a copy of the block, and TestPipelines seems to pass now. I also added some extra debug logging just to illustrate the problem.
This probably should not be committed until we can add another test which shows the problem 100% of the time.