[SPARK-49462] A query running under ckpt version v1 should be able to read a STATELESS query running under ckptv2

• Type: Task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 4.0.0
• Component/s: SS
• Epic Link: RocksDB State Store Checkpoint Structure V2
• Assignee: Unassigned
• Reporter: Wei Liu
• Created: 29/Aug/24 18:22
• Updated: 29/Aug/24 18:22

Description

Follow up to https://github.com/apache/spark/pull/47932

The validate version method force checks if the version persisted in the commit log 

https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/CommitLog.scala#L59

 

But the V2 checkpoint is only meaningful to stateful queries, so for a stateless query, a CommitLog running under V1 (old code) should be able to process a persisted commit log that has V2 format (new log)