This bug was revealed by the following system test failure http://confluent-systest.s3-website-us-west-2.amazonaws.com/confluent-kafka-system-test-results/?prefix=2017-07-18--001.1500383975--apache--trunk--28c83d9/
What happened was that a commit marker to the offsets topic (sent as part of the producer.sendOffsetsToTransaction method) was lost, causing data to be reprocessed, and hence causing the test to fail.
The bug is that the wrong error code is returned from the handleWriteTxnMarker request when there is partition emigration. In particular, we have:
But the replicaManager.getMagic() call will return None when the partition emigrates, causing the handleWriteTxnMarkersRequest to return an UNSUPPORTED_FOR_MESSAGE_FORMAT error.
From the log, we see that the partition did emigrate a few milliseconds before the WriteTxnMarkerRequest was sent.
On the old leader, worker10:
On the client:
As you can see, the client received the response 9 ms after the emigration was initiated on the server.
Since it is perfectly acceptable for the LeaderISR metadata to be propagated asynchronously, we should have more robust handling of emgiration in KafkaApis so that it returns the right error code when handling a request for a partition for which the broker is no longer the leader.