[SPARK-19779] structured streaming exist needless tmp file

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.0.3, 2.1.1, 2.2.0
• Fix Version/s: 2.0.3, 2.1.1, 2.2.0
• Component/s: Structured Streaming
• Assignee: Feng Gui
• Reporter: Feng Gui
• Created: 01/Mar/17 04:50
• Updated: 03/Mar/17 05:21
• Resolved: 03/Mar/17 05:20

Description

The PR (https://github.com/apache/spark/pull/17012) can to fix restart a Structured Streaming application using hdfs as fileSystem, but also exist a problem that a tmp file of delta file is still reserved in hdfs. And Structured Streaming don't delete the tmp file generated when restart streaming job in future, so we need to delete the tmp file after restart streaming job.

Issue Links

• is related to: SPARK-19677 HDFSBackedStateStoreProvider fails to overwrite existing file
• links to: [Github] Pull Request #17124 (gf53520)