[STORM-2138] java.io.FileNotFoundException: stormconf.ser does not exist

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.0.2
• Component/s: storm-core
• Assignee: Unassigned
• Reporter: Eddy
• Created: 07/Oct/16 13:09
• Updated: 17/Sep/18 09:31

Description

We are seeing problems in our storm topology whereby all our workers crash.

The errors we see are

2016-10-07 09:49:33.599 o.a.s.d.supervisor [ERROR] Error on initialization of server mk-supervisor
java.io.FileNotFoundException: File '/opt/storm_local/supervisor/stormdist/production_2016_09_13-1-1475831938/stormconf.ser' does not exist
at org.apache.storm.shade.org.apache.commons.io.FileUtils.openInputStream(FileUtils.java:292)
at org.apache.storm.shade.org.apache.commons.io.FileUtils.readFileToByteArray(FileUtils.java:1815)
at org.apache.storm.config$read_supervisor_storm_conf_given_path.invoke(config.clj:142)
at org.apache.storm.config$read_supervisor_storm_conf.invoke(config.clj:221)
at org.apache.storm.daemon.supervisor$add_blob_references.invoke(supervisor.clj:495)
at org.apache.storm.daemon.supervisor$fn_9307$exec_fn2466auto___9308.invoke(supervisor.clj:795)
at clojure.lang.AFn.applyToHelper(AFn.java:160)
at clojure.lang.AFn.applyTo(AFn.java:144)
at clojure.core$apply.invoke(core.clj:630)
at org.apache.storm.daemon.supervisor$fn_9307$mk_supervisor_9352.doInvoke(supervisor.clj:763)
at clojure.lang.RestFn.invoke(RestFn.java:436)
at org.apache.storm.daemon.supervisor$_launch.invoke(supervisor.clj:1200)
at org.apache.storm.daemon.supervisor$_main.invoke(supervisor.clj:1233)
at clojure.lang.AFn.applyToHelper(AFn.java:152)
at clojure.lang.AFn.applyTo(AFn.java:144)
at org.apache.storm.daemon.supervisor.main(Unknown Source)
2016-10-07 09:49:33.608 o.a.s.util [ERROR] Halting process: ("Error on initialization")
java.lang.RuntimeException: ("Error on initialization")
at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341)
at clojure.lang.RestFn.invoke(RestFn.java:423)
at org.apache.storm.daemon.supervisor$fn_9307$mk_supervisor_9352.doInvoke(supervisor.clj:763)
at clojure.lang.RestFn.invoke(RestFn.java:436)
at org.apache.storm.daemon.supervisor$_launch.invoke(supervisor.clj:1200)
at org.apache.storm.daemon.supervisor$_main.invoke(supervisor.clj:1233)
at clojure.lang.AFn.applyToHelper(AFn.java:152)
at clojure.lang.AFn.applyTo(AFn.java:144)
at org.apache.storm.daemon.supervisor.main(Unknown Source)
2016-10-07 09:49:34.668 o.a.s.d.supervisor [INFO] Removing code for storm id production_2016_09_13-1-1475831938

We have looked at https://github.com/apache/storm/pull/418 and https://issues.apache.org/jira/browse/STORM-130, which both show the first issue as being fixed - however we are still experiencing it in 1.0.2. The changes from the fixing commit (https://github.com/apache/storm/pull/418/commits/ccd28f8a356f468e66865fa9d9901b0a2628ec74) don't seem to be in the current version of the file (https://github.com/apache/storm/blob/v1.0.2/storm-core/src/clj/org/apache/storm/daemon/supervisor.clj).

We get this often when resubmitting a topology, and our only workaround is to stop the topology, delete the whole /opt/storm_local directory (which is our storm.local.dir) and resubmit the topology. Often, the workers seem to be looking for stormconf.ser in the local directory of an old topology that isn't even running at the time.