Description
In StreamingContext.stop(), when an exception is thrown the rest of the stop/cleanup action is aborted.
Discussed in https://github.com/apache/spark/pull/9116,
srowen commented
Hm, this is getting unwieldy. There are several nested try blocks here. The same argument goes for many of these methods – if one fails should they not continue trying? A more tidy solution would be to execute a series of () -> Unit code blocks that perform some cleanup and make sure that they each fire in succession, regardless of the others. The final one to remove the shutdown hook could occur outside synchronization.
I realize we're expanding the scope of the change here, but is it maybe worthwhile to go all the way here?
Really, something similar could be done for SparkContext and there's an existing JIRA for it somewhere.
At least, I'd prefer to either narrowly fix the deadlock here, or fix all of the finally-related issue separately and all at once.
Attachments
Issue Links
- duplicates
-
SPARK-11139 Make SparkContext.stop() exception-safe
-
- Resolved
-
- is duplicated by
-
-
- Resolved
-
- links to
