[SPARK-29953] File stream source cleanup options may break a file sink output

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: Structured Streaming
• Assignee: Jungtaek Lim
• Reporter: Shixiong Zhu
• Created: 18/Nov/19 23:14
• Updated: 06/Dec/19 05:55
• Resolved: 06/Dec/19 05:48

Description

SPARK-20568 added options to file streaming source to clean up processed files. However, when applying these options to a directory that was written by a file streaming sink, it will make the directory not queryable any more because we delete files from the directory but they are still tracked by file sink logs.

I think we should block the options if the input source is a file streaming sink path (has "_spark_metadata" folder).

Issue Links

• is caused by: SPARK-20568 Delete files after processing in structured streaming
• links to: GitHub Pull Request #26590