Major Description
We have observed in our production environment that during Spark shutdown, if there are some active tasks, sometimes they will complete with incorrect results. We've tracked down the issue to a PythonRunner where it is returning partial result instead of throwing exception during Spark shutdown.
I think the better way to handle this is to have these tasks fail instead of complete with partial results (complete with partial is always bad IMHO)
