[STORM-2275] Nimbus crashed during state transition of topology

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 2.0.0
• Component/s: storm-core
• Assignee: Raghav Kumar Gautam
• Reporter: Raghav Kumar Gautam
• Created: 04/Jan/17 23:21
• Updated: 06/Jan/17 09:14
• Resolved: 06/Jan/17 09:14

Description

I am copying last few lines of the nimbus logs including stack trace.

2017-01-04 22:18:10.106 pool-15-thread-47 o.a.s.d.n.Nimbus [INFO] Activating DemoTest: DemoTest-21-1483568289
2017-01-04 22:18:11.646 timer o.a.s.s.EvenScheduler [INFO] Available slots: [f0ea57ab-86d6-401f-9429-52f479b1d69f:6704, f0ea57ab-86d6-401f-9429-52f479b1d69f:6705, f0ea57ab-86d6-401f-9429-52f479b1d69f:670\
6, f0ea57ab-86d6-401f-9429-52f479b1d69f:6707, f0ea57ab-86d6-401f-9429-52f479b1d69f:6708, f0ea57ab-86d6-401f-9429-52f479b1d69f:6709, f0ea57ab-86d6-401f-9429-52f479b1d69f:6700, f0ea57ab-86d6-401f-9429-52f4\
79b1d69f:6701, f0ea57ab-86d6-401f-9429-52f479b1d69f:6702, f0ea57ab-86d6-401f-9429-52f479b1d69f:6703]
2017-01-04 22:18:11.648 timer o.a.s.d.n.Nimbus [INFO] Setting new assignment for topology id DemoTest-21-1483568289: Assignment(master_code_dir:storm-local, node_host:{f0ea57ab-86d6-401f-9429-52f479b1d69\
f=node1}, executor_node_port:{[10, 10]=NodeInfo(node:f0ea57ab-86d6-401f-9429-52f479b1d69f, port:[6700]), [14, 14]=NodeInfo(node:f0ea57ab-86d6-401f-9429-52f479b1d69f, port:[6701]), [16, 16]=NodeInfo(node:\
f0ea57ab-86d6-401f-9429-52f479b1d69f, port:[6700]), [12, 12]=NodeInfo(node:f0ea57ab-86d6-401f-9429-52f479b1d69f, port:[6702]), [8, 8]=NodeInfo(node:f0ea57ab-86d6-401f-9429-52f479b1d69f, port:[6701]), [6,\
 6]=NodeInfo(node:f0ea57ab-86d6-401f-9429-52f479b1d69f, port:[6702]), [20, 20]=NodeInfo(node:f0ea57ab-86d6-401f-9429-52f479b1d69f, port:[6701]), [4, 4]=NodeInfo(node:f0ea57ab-86d6-401f-9429-52f479b1d69f,\
 port:[6700]), [2, 2]=NodeInfo(node:f0ea57ab-86d6-401f-9429-52f479b1d69f, port:[6701]), [18, 18]=NodeInfo(node:f0ea57ab-86d6-401f-9429-52f479b1d69f, port:[6702]), [11, 11]=NodeInfo(node:f0ea57ab-86d6-401\
f-9429-52f479b1d69f, port:[6701]), [15, 15]=NodeInfo(node:f0ea57ab-86d6-401f-9429-52f479b1d69f, port:[6702]), [7, 7]=NodeInfo(node:f0ea57ab-86d6-401f-9429-52f479b1d69f, port:[6700]), [9, 9]=NodeInfo(node\
:f0ea57ab-86d6-401f-9429-52f479b1d69f, port:[6702]), [21, 21]=NodeInfo(node:f0ea57ab-86d6-401f-9429-52f479b1d69f, port:[6702]), [5, 5]=NodeInfo(node:f0ea57ab-86d6-401f-9429-52f479b1d69f, port:[6701]), [3\
, 3]=NodeInfo(node:f0ea57ab-86d6-401f-9429-52f479b1d69f, port:[6702]), [19, 19]=NodeInfo(node:f0ea57ab-86d6-401f-9429-52f479b1d69f, port:[6700]), [17, 17]=NodeInfo(node:f0ea57ab-86d6-401f-9429-52f479b1d6\
9f, port:[6701]), [1, 1]=NodeInfo(node:f0ea57ab-86d6-401f-9429-52f479b1d69f, port:[6700]), [13, 13]=NodeInfo(node:f0ea57ab-86d6-401f-9429-52f479b1d69f, port:[6700])}, executor_start_time_secs:{[12, 12]=1\
483568291, [6, 6]=1483568291, [18, 18]=1483568291, [2, 2]=1483568291, [8, 8]=1483568291, [14, 14]=1483568291, [16, 16]=1483568291, [20, 20]=1483568291, [4, 4]=1483568291, [10, 10]=1483568291, [9, 9]=1483\
568291, [3, 3]=1483568291, [15, 15]=1483568291, [21, 21]=1483568291, [5, 5]=1483568291, [11, 11]=1483568291, [13, 13]=1483568291, [17, 17]=1483568291, [19, 19]=1483568291, [1, 1]=1483568291, [7, 7]=14835\
68291}, worker_resources:{NodeInfo(node:f0ea57ab-86d6-401f-9429-52f479b1d69f, port:[6702])=WorkerResources(mem_on_heap:0.0, mem_off_heap:0.0, cpu:0.0), NodeInfo(node:f0ea57ab-86d6-401f-9429-52f479b1d69f,\
 port:[6701])=WorkerResources(mem_on_heap:0.0, mem_off_heap:0.0, cpu:0.0), NodeInfo(node:f0ea57ab-86d6-401f-9429-52f479b1d69f, port:[6700])=WorkerResources(mem_on_heap:0.0, mem_off_heap:0.0, cpu:0.0)})
2017-01-04 22:18:11.660 timer o.a.s.d.n.Nimbus [INFO] Cleaning up DemoTest-20-1483567429
2017-01-04 22:18:11.668 timer o.a.s.d.n.Nimbus [INFO] Removing dependency jars from blobs - []
2017-01-04 22:18:12.420 pool-15-thread-51 o.a.s.d.n.Nimbus [INFO] Created download session for DemoTest-21-1483568289-stormjar.jar
2017-01-04 22:18:12.990 pool-15-thread-38 o.a.s.d.n.Nimbus [INFO] Created download session for DemoTest-21-1483568289-stormcode.ser
2017-01-04 22:18:12.995 pool-15-thread-59 o.a.s.d.n.Nimbus [INFO] Created download session for DemoTest-21-1483568289-stormconf.ser
2017-01-04 22:18:20.303 timer o.a.s.d.n.Nimbus [INFO] TRANSITION: DemoTest-20-1483567429 REMOVE null false
2017-01-04 22:18:20.304 timer o.a.s.d.n.Nimbus [ERROR] Error while processing event
java.lang.RuntimeException: java.lang.NullPointerException
        at org.apache.storm.daemon.nimbus.Nimbus.lambda$delayEvent$16(Nimbus.java:1174)
        at org.apache.storm.StormTimer$StormTimerTask.run(StormTimer.java:83)
Caused by: java.lang.NullPointerException
        at org.apache.storm.daemon.nimbus.Nimbus.transition(Nimbus.java:1215)
        at org.apache.storm.daemon.nimbus.Nimbus.lambda$delayEvent$16(Nimbus.java:1172)
        ... 1 more
2017-01-04 22:18:20.304 timer o.a.s.u.Utils [ERROR] Halting process: Error while processing event
java.lang.RuntimeException: Halting process: Error while processing event
        at org.apache.storm.utils.Utils.exitProcess(Utils.java:1792)
        at org.apache.storm.daemon.nimbus.Nimbus.lambda$new$15(Nimbus.java:1107)
        at org.apache.storm.StormTimer$StormTimerTask.run(StormTimer.java:104)
2017-01-04 22:18:20.315 Thread-9 o.a.s.d.n.Nimbus [INFO] Shutting down master

The problem is that we are assuming that the base will be non-null which is incorrect leading to NPE.

Issue Links

• is related to: STORM-2268 Fix integration test for Travis CI build
• links to: GitHub Pull Request #1857