[SPARK-20883] Improve StateStore APIs for efficiency

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.2.0
• Fix Version/s: 2.3.0
• Component/s: Structured Streaming
• Target Version/s: 2.3.0
• Assignee: Tathagata Das
• Reporter: Tathagata Das
• Created: 25/May/17 11:04
• Updated: 15/Mar/18 05:20
• Resolved: 30/May/17 22:33

Description

Current state store API has a bunch of problems that causes too many transient objects causing memory pressure.

• StateStore.get() returns Options which forces creation of Some/None objects for every get
• StateStore.iterator() returns tuples which forces creation of new tuple for each record returned
• StateStore.updates() requires the implementation to keep track of updates, while this is used minimally (only by Append mode in streaming aggregations). This can be totally removed.

Issue Links

• links to: [Github] Pull Request #18107 (tdas)