[SPARK-21192] Preserve State Store provider class configuration across StreamingQuery restarts

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 2.3.0
• Component/s: Structured Streaming
• Target Version/s: 2.3.0
• Assignee: Tathagata Das
• Reporter: Tathagata Das
• Created: 23/Jun/17 10:19
• Updated: 23/Jun/17 17:55
• Resolved: 23/Jun/17 17:55

Description

If the SQL conf for StateStore provider class is changed between restarts (i.e. query started with providerClass1 and attempted to restart using providerClass2), then the query will fail in a unpredictable way as files saved by one provider class cannot be used by the newer one.

Ideally, the provider class used to start the query should be used to restart the query, and the configuration in the session where it is being restarted should be ignored.

Issue Links

• links to: [Github] Pull Request #18402 (tdas)