[KAFKA-5339] Transactions system test with hard broker bounces fails sporadically

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.11.0.0
• Component/s: clients, core, producer
• Labels: exactly-once
• Assignee: Apurva Mehta
• Reporter: Apurva Mehta
• Created: 28/May/17 00:23
• Updated: 06/Jun/17 22:25
• Resolved: 06/Jun/17 22:25

Description

The transactions hard bounce test occasionally fails because the transactional message copy just seems to hang. In one of the client logs, I noticed:

[2017-05-27 20:36:12,596] WARN Got error produce response with correlation id 124 on topic-partition output-topic-0, retrying (2147483646 attempts left). Error: NOT_LEADER_FOR_PARTITION (org.apache.kafka.clients.producer.internals.Sender)
[2017-05-27 20:36:15,386] ERROR Uncaught error in kafka producer I/O thread:  (org.apache.kafka.clients.producer.internals.Sender)
java.lang.NullPointerException
        at org.apache.kafka.clients.producer.internals.TransactionManager$1.compare(TransactionManager.java:146)
        at org.apache.kafka.clients.producer.internals.TransactionManager$1.compare(TransactionManager.java:143)
        at java.util.PriorityQueue.siftDownUsingComparator(PriorityQueue.java:721)
        at java.util.PriorityQueue.siftDown(PriorityQueue.java:687)
        at java.util.PriorityQueue.poll(PriorityQueue.java:595)
        at org.apache.kafka.clients.producer.internals.TransactionManager.nextRequestHandler(TransactionManager.java:351)
        at org.apache.kafka.clients.producer.internals.Sender.maybeSendTransactionalRequest(Sender.java:303)
        at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:193)
        at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:154)
        at java.lang.Thread.run(Thread.java:748)
[2017-05-27 20:36:52,007] INFO Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms. (org.apache.kafka.clients.producer.KafkaProducer)
[2017-05-27 20:36:52,036] INFO Marking the coordinator knode02:9092 (id: 2147483645 rack: null) dead for group transactions-test-consumer-group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
root@7dcd60017519:/opt/kafka-dev/results/latest/TransactionsTest/test_transactions/failure_mode=hard_bounce.bounce_target=brokers/1#

This suggests that the client has gotten to a bad state which is why it stops processing messages, causing the tests to fail.