Details
-
Bug
-
Status: Closed
-
Minor
-
Resolution: Abandoned
-
None
-
None
-
$ java -version
java version "1.7.0_45"
Java(TM) SE Runtime Environment (build 1.7.0_45-b18)
Java HotSpot(TM) 64-Bit Server VM (build 24.45-b08, mixed mode)
Description
We've been testing some Storm topologies using "lein repl". Based on the storm-starter project, our local testing environment offers a Clojure function (run-local!), which builts up the topology using the Clojure DSL and runs it. The issue is that there are conditions that can cause topology to fail; the exception thrown then crashes the Storm worker, which, in turn, crashes the lein repl itself. This is frustrating since part of the purpose of running a local cluster is to test it from something like a REPL.
I think I've narrowed down what's going wrong. You can see an example session with stacktrace here:
https://gist.github.com/amontalenti/8677464#file-example_storm_crashing_lein-txt-L142-L173
The way I created this error is by renaming my Python module before running the topology, so that the Python file could not be found.
I think what's going on is that the ShellBolt is throwing a RuntimeException, which is uncaught by whatever is running the ShellBolt. This, in turn, crashes the worker: "Error when launching multilang subprocess". The executor notices that the worker dies, but is a bit zealous and decides to call (halt-process!) on it. Under the hood, halt-process! uses Runtime.getRuntime#halt, which is a forcible kill of the running JVM. Since the JVM, in this case, is "lein repl", I think this is what ultimately kills the REPL.
http://docs.oracle.com/javase/7/docs/api/java/lang/Runtime.html#halt(int)
I suppose there are two possible fixes here. One is to make the Storm worker a little more resilient to a misconfigured ShellBolt. The other is to make the halt-process! call not run when a REPL environment is detected.
I am glad to work on either of these fixes once the issue is confirmed and a path forward is suggested.
