[SPARK-20376] Make StateStoreProvider plugable

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.1.0
• Fix Version/s: 2.3.0
• Component/s: Structured Streaming
• Target Version/s: 2.3.0
• Assignee: Tathagata Das
• Reporter: Yogesh Mahajan
• Created: 18/Apr/17 21:52
• Updated: 07/Feb/20 17:23
• Resolved: 13/Jul/17 18:20

Description

Currently only option available for StateStore and StateStoreProvider is HDFSBackedStateStoreProvider. There is no way to provide your own custom implementation of StateStoreProvider for users of structured streaming. We have built a performant in memory StateStore which we would like to plugin for our use cases.

Issue Links

• links to: [Github] Pull Request #18107 (tdas)