[SPARK-3912] FlumeStreamSuite is flaky, fails either with port binding issues or data not being reliably sent

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.1.0
• Fix Version/s: 1.2.0
• Component/s: DStreams
• Labels: flaky-test
• Target Version/s: 1.2.0
• Assignee: Tathagata Das
• Reporter: Tathagata Das
• Created: 12/Oct/14 00:19
• Updated: 19/Dec/14 16:13
• Resolved: 14/Oct/14 05:47

Description

Two problems.

1. Attempts to start the service to start on different possible ports (to avoid bind failures) was incorrect as the service is actually start lazily (when receiver starts, not when the flume input stream is created).
2. Lots of Thread.sleep was used to improve the probabilities that data sent through avro to flume receiver was being sent. However, the sending may fail for various unknown reasons, causing the test to fail.
3. Thread.sleep was also used to send one record per batch and checks were made on whether only one records was received in every batch. This was an overkill because all we need to test in this unit test is whether data is being sent and received or not, not about timings.

Issue Links

• links to: [Github] Pull Request #2773 (tdas)