[SPARK-39589] Asynchronous I/O API support in Structured Streaming

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.3.0
• Component/s: Structured Streaming
• Assignee: Unassigned
• Reporter: Boyang Jerry Peng
• Created: 24/Jun/22 20:58
• Updated: 12/Dec/22 18:11

Description

Often in streaming pipelines used to support ETL use cases, API calls to external systems can be found.  To better support and improve performance for such use cases, an asynchronous processing API should be added to Structured Streaming so that external requests can be handled in a more efficient manner.

Issue Links

• Is contained by: SPARK-40025 Project Lightspeed: Faster and Simpler Stream Processing with Apache Spark