[KAFKA-4527] Transient failure of ConnectDistributedTest.test_pause_and_resume_sink where paused connector produces messages

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.10.2.0
• Component/s: connect, system tests
• Labels: system-test-failure system-tests
• Assignee: Shikhar Bhushan
• Reporter: Ewen Cheslack-Postava
• Reviewer: Ewen Cheslack-Postava
• Created: 12/Dec/16 21:57
• Updated: 29/Dec/16 18:11
• Resolved: 24/Dec/16 22:24

Description

====================================================================================================
test_id: kafkatest.tests.connect.connect_distributed_test.ConnectDistributedTest.test_pause_and_resume_sink
status: FAIL
run time: 40.164 seconds

Paused sink connector should not consume any messages
Traceback (most recent call last):
File "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.6.0-py2.7.egg/ducktape/tests/runner_client.py", line 123, in run
data = self.run_test()
File "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.6.0-py2.7.egg/ducktape/tests/runner_client.py", line 176, in run_test
return self.test_context.function(self.test)
File "/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/tests/connect/connect_distributed_test.py", line 257, in test_pause_and_resume_sink
assert num_messages == len(self.sink.received_messages()), "Paused sink connector should not consume any messages"
AssertionError: Paused sink connector should not consume any messages

See one case here: http://confluent-kafka-system-test-results.s3-us-west-2.amazonaws.com/2016-12-12--001.1481535295--apache--trunk--62e043a/report.html but it has also happened before, e.g. http://confluent-kafka-system-test-results.s3-us-west-2.amazonaws.com/2016-12-06--001.1481017508--apache--trunk--34aa538/report.html

Thinking about the test, one simple possibility is that our approach to get the number of messages produced/consumed during the test is flawed – I think we may not account for additional buffering between the connectors and the process reading their output to determine what they have produced. However, that's just a theory – the minimal checking on the logs that I did didn't reveal anything obviously wrong.

Issue Links

• relates to: KAFKA-4575 Transient failure in ConnectDistributedTest.test_pause_and_resume_sink in consuming messages after resuming sink connector
• links to: GitHub Pull Request #2277