[SPARK-46705] Reduce RocksDB flush delay during checkpoint

• Type: Task
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 4.0.0
• Component/s: Structured Streaming
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: Siying Dong
• Created: 12/Jan/24 20:25
• Updated: 06/May/24 00:19

Description

When checkpointing a full snapshot for RocksDB, we do a flush. Even if the checkpointing is async, the flush is blocking the next batch from running. We need to reduce the delay in this flush.

Issue Links

• links to: GitHub Pull Request #44712