[SPARK-18342] HDFSBackedStateStore can fail to rename files causing snapshotting and recovery to fail

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.0.1
• Fix Version/s: 2.0.2, 2.1.0
• Component/s: Structured Streaming
• Target Version/s: 2.1.0
• Assignee: Burak Yavuz
• Reporter: Burak Yavuz
• Created: 08/Nov/16 04:04
• Updated: 10/Nov/16 13:29
• Resolved: 08/Nov/16 23:09

Description

The HDFSBackedStateStore renames temporary files to delta files as it commits new versions. It however doesn't check whether the rename succeeded. If the rename fails, then recovery will not be possible. It should fail during the rename stage.

Issue Links

• links to: [Github] Pull Request #15804 (brkyvz)