[SPARK-29999] Writing empty partition via DSv2 implementation of FileStreamSink throws FileNotFoundException

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: SQL
• Assignee: Jungtaek Lim
• Reporter: Jungtaek Lim
• Created: 22/Nov/19 11:16
• Updated: 26/Jun/20 07:23
• Resolved: 24/Nov/19 23:31

Description

I've discovered this issue while playing with Spark 3.0.0-preview.

Unlike DSv1 version of file sink, DSv2 version of file sink lazily creates output file which leads empty partition to not actually create output file. The behavior itself is not a problem, but while committing task in ManifestFileCommitProtocol, `fs.getFileStatus` throws FileNotFoundException since the file is not created.

 

Issue Links

• links to: [Github] Pull Request #28930 (HeartSaVioR)
• links to: [Github] Pull Request #28930 (HeartSaVioR)
• links to: GitHub Pull Request #26639