[SPARK-26872] Use a configurable value for final termination in the JobScheduler.stop() method

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Won't Fix
• Affects Version/s: 2.4.0
• Component/s: Scheduler, Spark Core
• Assignee: Unassigned
• Reporter: Steven Rosenberry
• Created: 13/Feb/19 19:19
• Updated: 17/May/20 17:46
• Resolved: 28/May/19 15:52

Description

As a user of Spark, I would like to configure the timeout that controls final termination after stopping the streaming context and while processing previously queued jobs.  Currently, there is a hard-coded limit of one hour around line 129 in the JobScheduler.stop() method:

// Wait for the queued jobs to complete if indicated
val terminated = if (processAllReceivedData) {
jobExecutor.awaitTermination(1, TimeUnit.HOURS) // just a very large period of time
} else {
jobExecutor.awaitTermination(2, TimeUnit.SECONDS)
}

It would provide additional functionality to the Spark platform if this value was configurable.  My use case may take many hours to finish the queued job as it was created from a large data file.

Issue Links

• links to: GitHub Pull Request #23926