[SPARK-46731] Do not register state store provider into coordinator in state data source - reader

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: Structured Streaming
• Labels: pull-request-available
• Assignee: Jungtaek Lim
• Reporter: Jungtaek Lim
• Created: 16/Jan/24 07:19
• Updated: 19/Jan/24 23:12
• Resolved: 19/Jan/24 23:12

Description

Currently we initialize state store instance via StateStore.get() which initializes state store provider instance and registers the provider instance to the coordinator. This involves unnecessary overheads e.g. maintenance task could be triggered for this provider.

The state data source should just initialize the provider and manage the provider instance by itself.

Issue Links

• links to: GitHub Pull Request #44751