[SPARK-18416] State Store leaks temporary files

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.0, 2.0.1, 2.0.2
• Fix Version/s: 2.0.3, 2.1.0
• Component/s: Structured Streaming
• Target Version/s: 2.1.0
• Assignee: Tathagata Das
• Reporter: Tathagata Das
• Created: 12/Nov/16 01:02
• Updated: 14/Nov/16 18:04
• Resolved: 14/Nov/16 18:04

Description

StateStore.get() causes temporary files to be created immediately, even if the store is not used to make updates for new version. The temp file is not closed as store.commit() is not called in those cases, thus keeping the output stream to temp file open forever.

Issue Links

• links to: [Github] Pull Request #15859 (tdas)