[SPARK-39593] Configurable State Checkpointing Frequency in Structured Streaming

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.3.0
• Component/s: Structured Streaming
• Assignee: Unassigned
• Reporter: Boyang Jerry Peng
• Created: 24/Jun/22 21:01
• Updated: 12/Dec/22 18:11

Description

Currently, for stateful pipelines state changes are checkpointed for every micro-batch. State checkpoints can contribute significantly to the latency of a micro-batch.  If state is checkpointed less frequently, its effect on batch latency can be amortized.  This can be used in conjunction with asynchronous state checkpointing to further reduce the cost in latency state checkpointing may incur.

Issue Links

• Is contained by: SPARK-40025 Project Lightspeed: Faster and Simpler Stream Processing with Apache Spark