[SPARK-39591] SPIP: Asynchronous Offset Management in Structured Streaming

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.4.0
• Component/s: Structured Streaming
• Labels: SPIP
• Assignee: Boyang Jerry Peng
• Reporter: Boyang Jerry Peng
• Shepherd: Jungtaek Lim
• Created: 24/Jun/22 20:59
• Updated: 20/Dec/22 07:51
• Resolved: 20/Dec/22 07:39

Description

Currently in Structured Streaming, at the beginning of every micro-batch the offset to process up to for the current batch is persisted to durable storage.  At the end of every micro-batch, a marker to indicate the completion of this current micro-batch is persisted to durable storage. For pipelines such as one that read from Kafka and write to Kafka, end-to-end exactly once is not support and latency is sensitive, we can allow users to configure offset commits to be written asynchronously thus this commit operation will not contribute to the batch duration and effectively lowering the overall latency of the pipeline.

 

SPIP Doc: 

 

https://docs.google.com/document/d/1iPiI4YoGCM0i61pBjkxcggU57gHKf2jVwD7HWMHgH-Y/edit?usp=sharing

Issue Links

• Is contained by: SPARK-40025 Project Lightspeed: Faster and Simpler Stream Processing with Apache Spark
• is related to: SPARK-41596 Document the new feature "Async Progress Tracking" to Structured Streaming guide doc
• links to: [Github] Pull Request #38517 (jerrypeng)