[STORM-1515] LocalState corruption after hard reboot on Windows

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.10.0, 0.9.4
• Fix Version/s: 2.1.0
• Component/s: storm-core
• Labels: pull-request-available
• Environment: Windows Server 2012
• Assignee: Tim Frison
• Reporter: Jordan Torbiak
• Created: 02/Feb/16 00:47
• Updated: 04/Sep/19 17:23
• Resolved: 04/Sep/19 17:23

Description

After a hard reboot on windows I'm seeing LocalState files for the supervisor that contain a few hundred NULs, resulting in a StreamCorruptedException on deserialization and the supervisor failing to start.

2016-01-27T17:04:10.848-0700 b.s.d.supervisor [INFO] Starting supervisor with id 45b27917-4ca0-4d96-8727-914909e3ac47 at host jtorbiak-ws.nj.invidi.com
2016-01-27T17:04:11.673-0700 b.s.event [ERROR] Error when processing event
java.lang.RuntimeException: java.io.StreamCorruptedException: invalid stream header: 00000000
        at backtype.storm.serialization.DefaultSerializationDelegate.deserialize(DefaultSerializationDelegate.java:56) ~[storm-core-0.9.4.jar:0.9.4]
        at backtype.storm.utils.Utils.deserialize(Utils.java:89) ~[storm-core-0.9.4.jar:0.9.4]
        at backtype.storm.utils.LocalState.deserializeLatestVersion(LocalState.java:65) ~[storm-core-0.9.4.jar:0.9.4]
        at backtype.storm.utils.LocalState.snapshot(LocalState.java:47) ~[storm-core-0.9.4.jar:0.9.4]
        at backtype.storm.utils.LocalState.get(LocalState.java:72) ~[storm-core-0.9.4.jar:0.9.4]
        at backtype.storm.daemon.supervisor$sync_processes.invoke(supervisor.clj:234) ~[storm-core-0.9.4.jar:0.9.4]
        at clojure.lang.AFn.applyToHelper(AFn.java:161) [clojure-1.5.1.jar:na]
        at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.5.1.jar:na]
        at clojure.core$apply.invoke(core.clj:619) ~[clojure-1.5.1.jar:na]
        at clojure.core$partial$fn__4190.doInvoke(core.clj:2396) ~[clojure-1.5.1.jar:na]
        at clojure.lang.RestFn.invoke(RestFn.java:397) ~[clojure-1.5.1.jar:na]
        at backtype.storm.event$event_manager$fn__2809.invoke(event.clj:40) ~[storm-core-0.9.4.jar:0.9.4]
        at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
Caused by: java.io.StreamCorruptedException: invalid stream header: 00000000
        at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:806) ~[na:1.8.0_45]
        at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299) ~[na:1.8.0_45]
        at backtype.storm.serialization.DefaultSerializationDelegate.deserialize(DefaultSerializationDelegate.java:51) ~[storm-core-0.9.4.jar:0.9.4]
        ... 13 common frames omitted
2016-01-27T17:04:11.674-0700 b.s.util [ERROR] Halting process: ("Error when processing an event")
java.lang.RuntimeException: ("Error when processing an event")
        at backtype.storm.util$exit_process_BANG_.doInvoke(util.clj:325) [storm-core-0.9.4.jar:0.9.4]
        at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.5.1.jar:na]
        at backtype.storm.event$event_manager$fn__2809.invoke(event.clj:48) [storm-core-0.9.4.jar:0.9.4]
        at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
2016-01-27T17:04:11.695-0700 b.s.d.supervisor [INFO] Shutting down supervisor 45b27917-4ca0-4d96-8727-914909e3ac47

This is very similar to STORM-307, except the LocalState files contain NULs instead of being empty. I'm guessing this corruption type is specific to Windows.

Issue Links

• links to: GitHub Pull Request #1067
• links to: GitHub Pull Request #3101