[STORM-582] Nimbus Halt with FileNotFoundException: '../nimbus/../stormconf.ser' dose not exists

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.9.2-incubating
• Component/s: storm-core
• Assignee: Harsha
• Reporter: Jiahong Li
• Created: 03/Dec/14 13:41
• Updated: 05/Oct/15 01:50
• Resolved: 22/Jun/15 04:58

Description

To notice, it is different from STORM-130. It is nimbus to halt. We ran into this problem several times, every time it happens is after several days of stable running. Here is the stacktrace
======================================================
2014-11-03 14:30:56 b.s.d.nimbus [INFO] Cleaning inbox ... deleted: stormjar-c1c856f0-cf8b-4299-9c20-712f169802b7.jar
2014-11-12 19:32:33 b.s.d.nimbus [ERROR] Error when processing event
java.io.FileNotFoundException: File '/tmp/storm-0.9.3/nimbus/stormdist/DNSAnalyse-7-1414992073/stormconf.ser' does not exist
at org.apache.commons.io.FileUtils.openInputStream(FileUtils.java:299) ~[commons-io-2.4.jar:2.4]
at org.apache.commons.io.FileUtils.readFileToByteArray(FileUtils.java:1763) ~[commons-io-2.4.jar:2.4]
at backtype.storm.daemon.nimbus$read_storm_conf.invoke(nimbus.clj:89) ~[storm-core-0.9.3-incubating-SNAPSHOT.jar:0.9.3-incubating-SNAPSHOT]
at backtype.storm.daemon.nimbus$read_topology_details.invoke(nimbus.clj:324) ~[storm-core-0.9.3-incubating-SNAPSHOT.jar:0.9.3-incubating-SNAPSHOT]
at backtype.storm.daemon.nimbus$mk_assignments$iter_31003104$fn_3105.invoke(nimbus.clj:649) ~[storm-core-0.9.3-incubating-SNAPSHOT.jar:0.9.3-incubating-SNAPSHOT]
at clojure.lang.LazySeq.sval(LazySeq.java:42) ~[clojure-1.5.1.jar:na]
at clojure.lang.LazySeq.seq(LazySeq.java:60) ~[clojure-1.5.1.jar:na]
at clojure.lang.RT.seq(RT.java:484) ~[clojure-1.5.1.jar:na]
at clojure.core$seq.invoke(core.clj:133) ~[clojure-1.5.1.jar:na]
at clojure.core.protocols$seq_reduce.invoke(protocols.clj:30) ~[clojure-1.5.1.jar:na]
at clojure.core.protocols$fn__6026.invoke(protocols.clj:54) ~[clojure-1.5.1.jar:na]
at clojure.core.protocols$fn_5979$G5974_5992.invoke(protocols.clj:13) ~[clojure-1.5.1.jar:na]
at clojure.core$reduce.invoke(core.clj:6177) ~[clojure-1.5.1.jar:na]
at clojure.core$into.invoke(core.clj:6229) ~[clojure-1.5.1.jar:na]
at backtype.storm.daemon.nimbus$mk_assignments.doInvoke(nimbus.clj:648) ~[storm-core-0.9.3-incubating-SNAPSHOT.jar:0.9.3-incubating-SNAPSHOT]
at clojure.lang.RestFn.invoke(RestFn.java:410) ~[clojure-1.5.1.jar:na]
at backtype.storm.daemon.nimbus$fn_3281$exec_fn1205auto__3282$fn3287$fn_3288.invoke(nimbus.clj:907) ~[storm-core-0.9.3-incubating-SNAPSHOT.jar:0.9.3-incubating-SNAPSHOT]
at backtype.storm.daemon.nimbus$fn_3281$exec_fn1205auto__3282$fn_3287.invoke(nimbus.clj:906) ~[storm-core-0.9.3-incubating-SNAPSHOT.jar:0.9.3-incubating-SNAPSHOT]
at backtype.storm.timer$schedule_recurring$this__2169.invoke(timer.clj:99) ~[storm-core-0.9.3-incubating-SNAPSHOT.jar:0.9.3-incubating-SNAPSHOT]
at backtype.storm.timer$mk_timer$fn_2152$fn_2153.invoke(timer.clj:50) ~[storm-core-0.9.3-incubating-SNAPSHOT.jar:0.9.3-incubating-SNAPSHOT]
at backtype.storm.timer$mk_timer$fn__2152.invoke(timer.clj:42) [storm-core-0.9.3-incubating-SNAPSHOT.jar:0.9.3-incubating-SNAPSHOT]
at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
at java.lang.Thread.run(Thread.java:679) [na:1.6.0_22]
2014-11-12 19:32:33 b.s.util [INFO] Halting process: ("Error when processing an event")
2014-11-12 19:32:33 b.s.d.nimbus [INFO] Shutting down master
=====================================================
To notice, after stable running for 9 days, without printing "Clean up

{storm-id}

" logs, nimbus halt.