[KAFKA-14263] Investigate flaky upgrade_test.py (missing messages)

• Type: Test
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.3.0
• Assignee: Unassigned
• Reporter: David Arthur
• Created: 27/Sep/22 18:26
• Updated: 27/Sep/22 18:26

Description

During system tests, we occasionally see flaky failures of upgrade_test.py.

Here is one such recent failure on the 3.3 branch:

Module: kafkatest.tests.core.upgrade_test
Class:  TestUpgrade
Method: test_upgrade
Arguments:
{
  "compression_types": [
    "none"
  ],
  "from_kafka_version": "0.9.0.1",
  "to_message_format_version": "0.9.0.1"
}

AssertionError("2 acked message did not make it to the Consumer. They are: [21017, 21014]. We validated that the first 2 of these missing messages correctly made it into Kafka's data files. This suggests they were lost on their way to the consumer.")

It would be good to try and reproduce these types of failures and see how we can make the test more robust.