[SPARK-25331] Structured Streaming File Sink duplicates records in case of driver failure

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.3.1
• Component/s: Structured Streaming
• Assignee: Unassigned
• Reporter: Mihaly Toth
• Created: 04/Sep/18 17:31
• Updated: 23/May/19 20:15
• Resolved: 23/May/19 20:15

Description

Lets assume FileStreamSink.addBtach is called and an appropriate job has been started by FileFormatWrite.write and then the resulting task sets are completed but in the meantime the driver dies. In such a case repeating FileStreamSink.addBtach will result in duplicate writing of the data

In the event the driver fails after the executors start processing the job the processed batch will be written twice.

Steps needed:

1. call FileStreamSink.addBtach
2. make the ManifestFileCommitProtocol fail to finish its commitJob
3. call FileStreamSink.addBtach with the same data
4. make the ManifestFileCommitProtocol finish its commitJob successfully
5. Verify file output - according to Sink.addBatch documentation the rdd should be written only once

I have created a wip PR with a unit test:
https://github.com/apache/spark/pull/22331

Issue Links

• duplicates: SPARK-19633 FileSource read from FileSink
• links to: [Github] Pull Request #22331 (misutoth)