Description
When the streaming job is configured with backpressureEnabled=true, it generates a GIANT batch of records if the processing time + scheduled delay is (much) larger than batchDuration. This creates a backlog of records like no other and results in batches queueing for hours until it chews through this giant batch.
Expectation is that it should reduce the number of records per batch in some time to whatever it can really process.
Attaching some screen shots where it seems that this issue is quite easily reproducible.
Attachments
Attachments
Issue Links
- links to