[KAFKA-3795] Transient system test failure upgrade_test.TestUpgrade

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: system tests
• Labels: kip-101 reliability
• Assignee: Unassigned
• Reporter: Jason Gustafson
• Created: 06/Jun/16 21:15
• Updated: 03/Apr/17 16:40

Description

From a recent build running on the 0.10.0 branch:

test_id:    2016-06-06--001.kafkatest.tests.core.upgrade_test.TestUpgrade.test_upgrade.from_kafka_version=0.9.0.1.to_message_format_version=0.9.0.1.compression_types=.snappy.new_consumer=True
status:     FAIL
run time:   3 minutes 29.166 seconds


    3522 acked message did not make it to the Consumer. They are: 476524, 476525, 476527, 476528, 476530, 476531, 476533, 476534, 476536, 476537, 476539, 476540, 476542, 476543, 476545, 476546, 476548, 476549, 476551, 476552, ...plus 3482 more. Total Acked: 110437, Total Consumed: 127470. The first 1000 missing messages were validated to ensure they are in Kafka's data files. 1000 were missing. This suggests data loss. Here are some of the messages not found in the data files: [477184, 477185, 477187, 477188, 477190, 477191, 477193, 477194, 477196, 477197]

Traceback (most recent call last):
  File "/var/lib/jenkins/workspace/system-test-kafka-0.10.0/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/tests/runner.py", line 106, in run_all_tests
    data = self.run_single_test()
  File "/var/lib/jenkins/workspace/system-test-kafka-0.10.0/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/tests/runner.py", line 162, in run_single_test
    return self.current_test_context.function(self.current_test)
  File "/var/lib/jenkins/workspace/system-test-kafka-0.10.0/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/mark/_mark.py", line 331, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/var/lib/jenkins/workspace/system-test-kafka-0.10.0/kafka/tests/kafkatest/tests/core/upgrade_test.py", line 113, in test_upgrade
    self.run_produce_consume_validate(core_test_action=lambda: self.perform_upgrade(from_kafka_version,
  File "/var/lib/jenkins/workspace/system-test-kafka-0.10.0/kafka/tests/kafkatest/tests/produce_consume_validate.py", line 79, in run_produce_consume_validate
    raise e
AssertionError: 3522 acked message did not make it to the Consumer. They are: 476524, 476525, 476527, 476528, 476530, 476531, 476533, 476534, 476536, 476537, 476539, 476540, 476542, 476543, 476545, 476546, 476548, 476549, 476551, 476552, ...plus 3482 more. Total Acked: 110437, Total Consumed: 127470. The first 1000 missing messages were validated to ensure they are in Kafka's data files. 1000 were missing. This suggests data loss. Here are some of the messages not found in the data files: [477184, 477185, 477187, 477188, 477190, 477191, 477193, 477194, 477196, 477197]

Here's a link to the test data: http://testing.confluent.io/confluent-kafka-0-10-0-system-test-results/?prefix=2016-06-06--001.1465234069--apache--0.10.0--6500b53/