[SPARK-27254] Cleanup complete but becoming invalid output files in ManifestFileCommitProtocol if job is aborted

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: Structured Streaming
• Assignee: Jungtaek Lim
• Reporter: Jungtaek Lim
• Created: 22/Mar/19 22:43
• Updated: 27/Sep/19 20:25
• Resolved: 27/Sep/19 19:37

Description

ManifestFileCommitProtocol doesn't clean up complete (but will become invalid) output files when job is aborted.

ManifestFileCommitProtocol doesn't do anything for cleaning up when job is aborted but just maintains the metadata which list of complete output files are written. SPARK-27210 addressed for task level cleanup, but it still doesn't clean up it as job level.

Issue Links

• links to: GitHub Pull Request #24186