[SPARK-26389] Add force delete temp checkpoint configuration

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 3.0.0
• Component/s: Structured Streaming
• Assignee: Gabor Somogyi
• Reporter: Fengyu Cao
• Created: 18/Dec/18 03:31
• Updated: 26/Dec/19 05:08
• Resolved: 08/Feb/19 18:24

Description

spark-submit --master mesos://<mesos> -conf spark.streaming.stopGracefullyOnShutdown=true <structured streaming framework>

CTRL-C, framework shutdown

18/12/18 10:27:36 ERROR MicroBatchExecution: Query [id = f512e17a-df88-4414-a5cd-a23550cf1e7f, runId = 24d99723-8d61-48c0-beab-af432f7a19d3] terminated with error org.apache.spark.SparkException: Writing job aborted.

/tmp/temporary-<uuid> on executor not deleted due to org.apache.spark.SparkException: Writing job aborted., and this temp checkpoint can't used to recovery.

 

Issue Links

• is duplicated by: SPARK-27634 deleteCheckpointOnStop should be configurable
• links to: GitHub Pull Request #23732
• links to: GitHub Pull Request #26981