[BAHIR-83] Flaky test in BasicMQTTSourceSuite

• Type: Test
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: Spark-2.0.1
• Component/s: Spark Structured Streaming Connectors
• Environment: Maven 3.3.9, JDK 1.8, 2015 MacBook Pro, 16 GB RAM, 50 GB free storage
• Assignee: Prashant Sharma
• Reporter: Christian Kadner
• Created: 04/Dec/16 08:52
• Updated: 07/Feb/20 17:23

Description

The test case "Recovering offset from the last processed offset." in BasicMQTTSourceSuite fails sporadically.

I did repeated runs of that particular test and it failed 6 of 100 runs.

for i in `seq 100` ; do
  mvn scalatest:test -pl sql-streaming-mqtt -q \
    -Dsuites='*.BasicMQTTSourceSuite @ Recovering offset from the last processed offset.' | \
    grep -q "TEST FAILED" && echo "$i: failed"
done
1: failed
6: failed
22: failed
46: failed
77: failed
83: failed

The complete test suite BasicMQTTSourceSuite failed 56/100 times.

EDIT 2017/01/13:

• the two test cases Send and receive 100 messages and Recovering offset from the last processed offset in BasicMQTTSourceSuite appear to fail when system resources are limited or under stress

• I was able to reproduce similar test failures by putting additional stress on CPU, memory and disk I/O using the stress tool

• when CPU-bound, the test cases can be modified to succeed by increasing the stream write timeout from 10 seconds to 15 or 20 seconds. However that increased timeout did not help when memory or I/O is limited

• it may be worth identifying and document minimum resource requirements for the SQL-MQTT connector. If we want to get sophisticated, that information could possibly be used to dynamically enable/disable test cases based on available system resources

Issue Links

• links to: Bahir dev mailing list - Vote 2.0.2 RC1 - SQL-MQTT Test Suite Failure
• links to: Bahir dev mailing list - Vote 2.0.2 RC2 - SQL-MQTT Test Suite Failure