[SPARK-16738] Queryable state for Spark State Store

• Type: New Feature
• Status: Reopened
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.0.0
• Component/s: Structured Streaming
• Labels: features
• Assignee: Unassigned
• Reporter: Mark Sumner
• Created: 26/Jul/16 16:34
• Updated: 12/Dec/22 18:10

Description

Spark 2.0 will introduce the new State Store to allow state managment outside the RDD model (see: SPARK-13809)

This proposal seeks to include a mechanism (in a future release) to expose this internal store to external applications for querying.

This would then make it possible to interact with aggregated state without needing to synchronously write (and read) to/from an external store.