[SPARK-42353] Cleanup orphan sst and log files in RocksDB checkpoint directory

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.3
• Fix Version/s: 3.5.0
• Component/s: Structured Streaming
• Assignee: Chaoqin Li
• Reporter: Chaoqin Li
• Created: 06/Feb/23 07:54
• Updated: 09/Feb/23 23:22
• Resolved: 09/Feb/23 23:22

Description

When RocksDB version.zip file get overwritten (e.g. concurrent task execution, task/stage/batch reattempts) or the zip file don't get uploaded successfully, the associated sst and log files don't get garbage collected.(https://github.com/databricks/runtime/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala) These files consume storage. We can clean up these SST files during periodic state store maintenance. The major concern is that sst files for ongoing version also appear to be "orphan" because they are uploaded before zip file, we have to be careful not to delete them.

Issue Links

• links to: [Github] Pull Request #39897 (chaoqin-li1123)