[KAFKA-5375] Transactions: Concurrent transactional consumer loses messages when there are broker bounces

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 0.11.0.0
• Labels: exactly-once
• Assignee: Apurva Mehta
• Reporter: Apurva Mehta
• Created: 03/Jun/17 21:10
• Updated: 06/Jun/17 20:52
• Resolved: 06/Jun/17 20:52

Description

With the system test modifications in KAFKA-5366, the concurrent reader almost always consumes a fraction of the expected messages when there are broker bounces. A consumer running without concurrent writes consumes all the messages from the topic in question in the same test.

================================================================================
test_id:    kafkatest.tests.core.transactions_test.TransactionsTest.test_transactions.failure_mode=clean_bounce.bounce_target=brokers
status:     FAIL
run time:   1 minute 59.169 seconds


    Input and concurrently consumed output message sets are not equal. Num input messages: 20000. Num concurrently_consumed_messages: 0
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/ducktape/tests/runner_client.py", line 123, in run
    data = self.run_test()
  File "/usr/local/lib/python2.7/dist-packages/ducktape/tests/runner_client.py", line 176, in run_test
    return self.test_context.function(self.test)
  File "/usr/local/lib/python2.7/dist-packages/ducktape/mark/_mark.py", line 321, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/opt/kafka-dev/tests/kafkatest/tests/core/transactions_test.py", line 238, in test_transactions
    (len(input_message_set), len(concurrently_consumed_message_set))
AssertionError: Input and concurrently consumed output message sets are not equal. Num input messages: 20000. Num concurrently_consumed_messages: 0