[SPARK-45472] RocksDB State Store Doesn't Need to Recheck checkpoint path existence

• Type: Task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.5.1
• Fix Version/s: 4.0.0
• Component/s: Structured Streaming
• Labels: pull-request-available
• Assignee: Siying Dong
• Reporter: Siying Dong
• Created: 09/Oct/23 17:56
• Updated: 10/Oct/23 02:24
• Resolved: 10/Oct/23 02:24

Description

Right now, every time RocksDB.load() is called, we check checkpoint directory existence and create it if not. This is relatively expensive and show up in performance profiling. We don't need to do it the second time.

Issue Links

• links to: GitHub Pull Request #43299