[SPARK-20209] Execute next trigger immediately if previous batch took longer than trigger interval

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.1.0
• Fix Version/s: 2.2.0
• Component/s: Structured Streaming
• Target Version/s: 2.2.0
• Assignee: Tathagata Das
• Reporter: Tathagata Das
• Created: 04/Apr/17 06:51
• Updated: 04/May/17 00:35
• Resolved: 05/Apr/17 06:17

Description

For large trigger intervals (e.g. 10 minutes), if a batch takes 11 minutes, then it will wait for 9 mins before starting the next batch. This does not make sense. The processing time based trigger policy should be to do process batches as fast as possible, but no faster than 1 in every trigger interval. If batches are taking longer than trigger interval anyways, then no point waiting extra trigger interval.

Issue Links

• is duplicated by: SPARK-20063 Trigger without delay when falling behind
• links to: [Github] Pull Request #17525 (tdas)