[SPARK-40025] Project Lightspeed: Faster and Simpler Stream Processing with Apache Spark

• Type: Umbrella
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.2.2
• Component/s: Structured Streaming
• Assignee: Unassigned
• Reporter: Boyang Jerry Peng
• Created: 09/Aug/22 22:47
• Updated: 04/Nov/22 17:36

Description

Project Lightspeed is an umbrella project aimed at improving a couple of key aspects of Spark Streaming:

• Improving the latency and ensuring it is predictable
• Enhancing functionality for processing data with new operators and APIs

 

Umbrella Jira to track all tickets under Project Lightspeed

SPARK-39585 - Multiple Stateful Operators in Structured Streaming
SPARK-39586 - Advanced Windowing in Structured Streaming
SPARK-39587 - Schema Evolution for Stateful Pipelines
SPARK-39589 - Asynchronous I/O support
SPARK-40431 - Python API for Arbitrary Stateful Processing
SPARK-39591 - Offset Management Improvements

SPARK-40849 - Async log purge

SPARK-39592 - Asynchronous State Checkpointing
SPARK-39593 - Configurable State Checkpointing Frequency

Issue Links

• contains: SPARK-39586 Advanced Windowing in Structured Streaming
• contains: SPARK-39589 Asynchronous I/O API support in Structured Streaming
• contains: SPARK-40849 Async log purge
• contains: SPARK-40957 Add in memory cache in HDFSMetadataLog
• contains: SPARK-39585 Multiple Stateful Operators in Structured Streaming
• contains: SPARK-39587 Schema Evolution for Stateful Pipelines in Structured Streaming
• contains: SPARK-39592 Asynchronous State Checkpointing in Structured Streaming
• contains: SPARK-39593 Configurable State Checkpointing Frequency in Structured Streaming
• contains: SPARK-39590 Python API Parity in Structure Streaming
• contains: SPARK-39591 SPIP: Asynchronous Offset Management in Structured Streaming
• Is contained by: SPARK-40957 Add in memory cache in HDFSMetadataLog