Running stages of Spark jobs submitted via Livy' programmatic API cannot (always) be successfully cancelled.
The current implementation of .JobWrapper.cancel() interrupts the worker thread on the Spark driver (via Future.cancel(true)):
This does not always cancel all activity in Spark, e.g. long-running stages may remain unaffected.
The Spark-way of cancelling jobs seems to be via SparkContext.setJobGroup()/cancelJobGroup(), which is also being used in Livy's REPL Session:
I have opened a PR that invokes setJobGroup()/cancelJobGroup() in addition to interrupting the worker thread running on the driver:
It would be great if the fix could make it into the 0.6 release.