[SPARK-45794] Introduce state metadata source to query the streaming state metadata information

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.5.0
• Fix Version/s: 4.0.0
• Component/s: Structured Streaming
• Labels: pull-request-available
• Assignee: Chaoqin Li
• Reporter: Chaoqin Li
• Created: 05/Nov/23 05:15
• Updated: 08/Nov/23 05:04
• Resolved: 08/Nov/23 05:04

Description

Introduce a new data source so that user can query the metadata of each state store of a streaming query, the schema of the result will be

operatorId INT | operatorName STRING | stateStoreName STRING | numPartitions INT | numColsPrefixKey INT | minBatchId  LONG | minBatchId LONG

To use this source, specify the source format and checkpoint path

df = spark.read.format(“state-metadata”).load(“/checkpointPath”)

Issue Links

• links to: GitHub Pull Request #43660