Description
Currently we have
1) StorageProxy.sendMessages() sending messages to the first node in the other DC...
2) A node in the other DC will remove the ForwardHeader and sendRR (Adding a MessageID to the Queue).
3) The receiving node receives the mutation, updates and sends the response to the Original Co-ordinator.
4) Co-Ordinator now checks for the MessageID (which it never had)
All the Quorum_Each updates fail in the co-ordinator, this issue started showing up after CASSANDRA-3472 the code was introduced in CASSANDRA-2138 .
Simple Fix is to remove the optimization in 0.8 and fix it in 1.x because it seems to me like it needs a change to the Message service version.
Possible Solution: We might want send the message ID's to be used by the all the nodes in other DC (Which is currently generated by the node which receives the Forward request see: (2) ).