[STORM-270] Why ahead-of-time compiled clojure "classes" are not used in storm-core

• Type: Question
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.9.2-incubating
• Component/s: storm-core
• Assignee: Robert Joseph Evans
• Reporter: Rafael Bagmanov
• Created: 30/Mar/14 19:05
• Updated: 09/Oct/15 00:47
• Resolved: 09/Apr/14 14:31

Description

Hi

I do see that storm-core.jar contains both *.clj sources and clojure compiler generated *.class files. But in my observation in storm UI daemon, every request triggers some clj compilation on the-flight.

This leads to:

• high latency. (it takes about 5 seconds to fully load storm UI index page)
• under very modest load (10 concurrent requests) index page rendering fails with exceptions for some requests:

java.lang.RuntimeException: Unable to resolve symbol: nimbus in this context, compiling:(backtype/storm/ui/core.clj:1038)
at clojure.lang.Compiler.analyze(Compiler.java:6281)
at clojure.lang.Compiler.analyze(Compiler.java:6223)
at clojure.lang.Compiler$InvokeExpr.parse(Compiler.java:3548)
at clojure.lang.Compiler.analyzeSeq(Compiler.java:6457)
at clojure.lang.Compiler.analyze(Compiler.java:6262)
at clojure.lang.Compiler.analyze(Compiler.java:6223)
at clojure.lang.Compiler$BodyExpr$Parser.parse(Compiler.java:5618)
at clojure.lang.Compiler$LetExpr$Parser.parse(Compiler.java:5919)
at clojure.lang.Compiler.analyzeSeq(Compiler.java:6455)
at clojure.lang.Compiler.analyze(Compiler.java:6262)
at clojure.lang.Compiler.analyzeSeq(Compiler.java:6443)
at clojure.lang.Compiler.analyze(Compiler.java:6262)
at clojure.lang.Compiler.analyzeSeq(Compiler.java:6443)
at clojure.lang.Compiler.analyze(Compiler.java:6262)
at clojure.lang.Compiler.analyze(Compiler.java:6223)
at clojure.lang.Compiler$BodyExpr$Parser.parse(Compiler.java:5618)
at clojure.lang.Compiler$FnMethod.parse(Compiler.java:5054)
at clojure.lang.Compiler$FnExpr.parse(Compiler.java:3674)
at clojure.lang.Compiler.analyzeSeq(Compiler.java:6453)
at clojure.lang.Compiler.analyze(Compiler.java:6262)
at clojure.lang.Compiler.analyzeSeq(Compiler.java:6443)
at clojure.lang.Compiler.analyze(Compiler.java:6262)
at clojure.lang.Compiler.analyze(Compiler.java:6223)
at clojure.lang.Compiler$InvokeExpr.parse(Compiler.java:3548)
at clojure.lang.Compiler.analyzeSeq(Compiler.java:6457)
at clojure.lang.Compiler.analyze(Compiler.java:6262)
at clojure.lang.Compiler.analyzeSeq(Compiler.java:6443)
at clojure.lang.Compiler.analyze(Compiler.java:6262)
at clojure.lang.Compiler.analyze(Compiler.java:6223)
at clojure.lang.Compiler$InvokeExpr.parse(Compiler.java:3548)
at clojure.lang.Compiler.analyzeSeq(Compiler.java:6457)
at clojure.lang.Compiler.analyze(Compiler.java:6262)
at clojure.lang.Compiler.access$100(Compiler.java:37)
at clojure.lang.Compiler$DefExpr$Parser.parse(Compiler.java:518)
at clojure.lang.Compiler.analyzeSeq(Compiler.java:6455)
at clojure.lang.Compiler.analyze(Compiler.java:6262)
at clojure.lang.Compiler.analyze(Compiler.java:6223)
at clojure.lang.Compiler.eval(Compiler.java:6515)
at clojure.lang.Compiler.load(Compiler.java:6952)
at clojure.lang.RT.loadResourceScript(RT.java:359)
at clojure.lang.RT.loadResourceScript(RT.java:350)
at clojure.lang.RT.load(RT.java:429)
at clojure.lang.RT.load(RT.java:400)
at clojure.core$load$fn__4890.invoke(core.clj:5415)
at clojure.core$load.doInvoke(core.clj:5414)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invoke(core.clj:5227)
at clojure.core$load_lib.doInvoke(core.clj:5264)
at clojure.lang.RestFn.applyTo(RestFn.java:142)
at clojure.core$apply.invoke(core.clj:603)
at clojure.core$load_libs.doInvoke(core.clj:5298)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invoke(core.clj:603)
at clojure.core$require.doInvoke(core.clj:5381)
at clojure.lang.RestFn.invoke(RestFn.java:421)
at ring.middleware.reload$wrap_reload$fn__7318.invoke(reload.clj:13)
at backtype.storm.ui.core$catch_errors$fn__2082.invoke(core.clj:1075)
at ring.middleware.keyword_params$wrap_keyword_params$fn__6327.invoke(keyword_params.clj:27)
at ring.middleware.nested_params$wrap_nested_params$fn__6364.invoke(nested_params.clj:65)
at ring.middleware.params$wrap_params$fn__6301.invoke(params.clj:55)
at ring.middleware.multipart_params$wrap_multipart_params$fn__6390.invoke(multipart_params.clj:103)
at ring.middleware.flash$wrap_flash$fn__6561.invoke(flash.clj:14)
at ring.middleware.session$wrap_session$fn__6552.invoke(session.clj:43)
at ring.middleware.cookies$wrap_cookies$fn__6489.invoke(cookies.clj:160)
at ring.adapter.jetty$proxy_handler$fn__7252.invoke(jetty.clj:16)
at ring.adapter.jetty.proxy$org.mortbay.jetty.handler.AbstractHandler$0.handle(Unknown Source)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: java.lang.RuntimeException: Unable to resolve symbol: nimbus in this context
at clojure.lang.Util.runtimeException(Util.java:170)
at clojure.lang.Compiler.resolveIn(Compiler.java:6766)
at clojure.lang.Compiler.resolve(Compiler.java:6710)
at clojure.lang.Compiler.analyzeSymbol(Compiler.java:6671)
at clojure.lang.Compiler.analyze(Compiler.java:6244)
... 74 more

I'm not sure whether same compilation processes happen in other storm daemons.

I've done experiments: after removing all clj files from the storm-core.jar, all issues described above disappeared: latency decreased to 800 ms, UI become able to handle 50 concurrent requests on the same machine.