[SPARK-18371] Spark Streaming backpressure bug - generates a batch with large number of records

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 2.4.0
• Component/s: DStreams
• Assignee: Sebastian Arzt
• Reporter: mapreduced
• Created: 09/Nov/16 00:26
• Updated: 29/Sep/19 13:14
• Resolved: 16/Mar/18 17:33

Description

When the streaming job is configured with backpressureEnabled=true, it generates a GIANT batch of records if the processing time + scheduled delay is (much) larger than batchDuration. This creates a backlog of records like no other and results in batches queueing for hours until it chews through this giant batch.
Expectation is that it should reduce the number of records per batch in some time to whatever it can really process.
Attaching some screen shots where it seems that this issue is quite easily reproducible.

Issue Links

• links to: [Github] Pull Request #17774 (arzt)