[SPARK-36533] Allow streaming queries with Trigger.Once run in multiple batches

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.3.0
• Component/s: Structured Streaming
• Labels: releasenotes
• Assignee: Bo Zhang
• Reporter: Bo Zhang
• Created: 17/Aug/21 09:48
• Updated: 14/Mar/22 23:41
• Resolved: 01/Sep/21 06:02

Description

Currently streaming queries with Trigger.Once will always load all of the available data in a single batch. Because of this, the amount of data the queries can process is limited, or Spark driver will be out of memory.

We should allow streaming queries with Trigger.Once run in multiple batches.

Issue Links

• links to: [Github] Pull Request #33763 (bozhang2820)
• links to: [Github] Pull Request #34592 (HeartSaVioR)
• links to: [Github] Pull Request #34592 (HeartSaVioR)
• links to: [Github] Pull Request #34597 (HeartSaVioR)