Uploaded image for project: 'Apache Storm'
  1. Apache Storm
  2. STORM-2275

Nimbus crashed during state transition of topology

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.0.0
    • Fix Version/s: 2.0.0
    • Component/s: storm-core
    • Labels:
      None

      Description

      I am copying last few lines of the nimbus logs including stack trace.

      2017-01-04 22:18:10.106 pool-15-thread-47 o.a.s.d.n.Nimbus [INFO] Activating DemoTest: DemoTest-21-1483568289
      2017-01-04 22:18:11.646 timer o.a.s.s.EvenScheduler [INFO] Available slots: [f0ea57ab-86d6-401f-9429-52f479b1d69f:6704, f0ea57ab-86d6-401f-9429-52f479b1d69f:6705, f0ea57ab-86d6-401f-9429-52f479b1d69f:670\
      6, f0ea57ab-86d6-401f-9429-52f479b1d69f:6707, f0ea57ab-86d6-401f-9429-52f479b1d69f:6708, f0ea57ab-86d6-401f-9429-52f479b1d69f:6709, f0ea57ab-86d6-401f-9429-52f479b1d69f:6700, f0ea57ab-86d6-401f-9429-52f4\
      79b1d69f:6701, f0ea57ab-86d6-401f-9429-52f479b1d69f:6702, f0ea57ab-86d6-401f-9429-52f479b1d69f:6703]
      2017-01-04 22:18:11.648 timer o.a.s.d.n.Nimbus [INFO] Setting new assignment for topology id DemoTest-21-1483568289: Assignment(master_code_dir:storm-local, node_host:{f0ea57ab-86d6-401f-9429-52f479b1d69\
      f=node1}, executor_node_port:{[10, 10]=NodeInfo(node:f0ea57ab-86d6-401f-9429-52f479b1d69f, port:[6700]), [14, 14]=NodeInfo(node:f0ea57ab-86d6-401f-9429-52f479b1d69f, port:[6701]), [16, 16]=NodeInfo(node:\
      f0ea57ab-86d6-401f-9429-52f479b1d69f, port:[6700]), [12, 12]=NodeInfo(node:f0ea57ab-86d6-401f-9429-52f479b1d69f, port:[6702]), [8, 8]=NodeInfo(node:f0ea57ab-86d6-401f-9429-52f479b1d69f, port:[6701]), [6,\
       6]=NodeInfo(node:f0ea57ab-86d6-401f-9429-52f479b1d69f, port:[6702]), [20, 20]=NodeInfo(node:f0ea57ab-86d6-401f-9429-52f479b1d69f, port:[6701]), [4, 4]=NodeInfo(node:f0ea57ab-86d6-401f-9429-52f479b1d69f,\
       port:[6700]), [2, 2]=NodeInfo(node:f0ea57ab-86d6-401f-9429-52f479b1d69f, port:[6701]), [18, 18]=NodeInfo(node:f0ea57ab-86d6-401f-9429-52f479b1d69f, port:[6702]), [11, 11]=NodeInfo(node:f0ea57ab-86d6-401\
      f-9429-52f479b1d69f, port:[6701]), [15, 15]=NodeInfo(node:f0ea57ab-86d6-401f-9429-52f479b1d69f, port:[6702]), [7, 7]=NodeInfo(node:f0ea57ab-86d6-401f-9429-52f479b1d69f, port:[6700]), [9, 9]=NodeInfo(node\
      :f0ea57ab-86d6-401f-9429-52f479b1d69f, port:[6702]), [21, 21]=NodeInfo(node:f0ea57ab-86d6-401f-9429-52f479b1d69f, port:[6702]), [5, 5]=NodeInfo(node:f0ea57ab-86d6-401f-9429-52f479b1d69f, port:[6701]), [3\
      , 3]=NodeInfo(node:f0ea57ab-86d6-401f-9429-52f479b1d69f, port:[6702]), [19, 19]=NodeInfo(node:f0ea57ab-86d6-401f-9429-52f479b1d69f, port:[6700]), [17, 17]=NodeInfo(node:f0ea57ab-86d6-401f-9429-52f479b1d6\
      9f, port:[6701]), [1, 1]=NodeInfo(node:f0ea57ab-86d6-401f-9429-52f479b1d69f, port:[6700]), [13, 13]=NodeInfo(node:f0ea57ab-86d6-401f-9429-52f479b1d69f, port:[6700])}, executor_start_time_secs:{[12, 12]=1\
      483568291, [6, 6]=1483568291, [18, 18]=1483568291, [2, 2]=1483568291, [8, 8]=1483568291, [14, 14]=1483568291, [16, 16]=1483568291, [20, 20]=1483568291, [4, 4]=1483568291, [10, 10]=1483568291, [9, 9]=1483\
      568291, [3, 3]=1483568291, [15, 15]=1483568291, [21, 21]=1483568291, [5, 5]=1483568291, [11, 11]=1483568291, [13, 13]=1483568291, [17, 17]=1483568291, [19, 19]=1483568291, [1, 1]=1483568291, [7, 7]=14835\
      68291}, worker_resources:{NodeInfo(node:f0ea57ab-86d6-401f-9429-52f479b1d69f, port:[6702])=WorkerResources(mem_on_heap:0.0, mem_off_heap:0.0, cpu:0.0), NodeInfo(node:f0ea57ab-86d6-401f-9429-52f479b1d69f,\
       port:[6701])=WorkerResources(mem_on_heap:0.0, mem_off_heap:0.0, cpu:0.0), NodeInfo(node:f0ea57ab-86d6-401f-9429-52f479b1d69f, port:[6700])=WorkerResources(mem_on_heap:0.0, mem_off_heap:0.0, cpu:0.0)})
      2017-01-04 22:18:11.660 timer o.a.s.d.n.Nimbus [INFO] Cleaning up DemoTest-20-1483567429
      2017-01-04 22:18:11.668 timer o.a.s.d.n.Nimbus [INFO] Removing dependency jars from blobs - []
      2017-01-04 22:18:12.420 pool-15-thread-51 o.a.s.d.n.Nimbus [INFO] Created download session for DemoTest-21-1483568289-stormjar.jar
      2017-01-04 22:18:12.990 pool-15-thread-38 o.a.s.d.n.Nimbus [INFO] Created download session for DemoTest-21-1483568289-stormcode.ser
      2017-01-04 22:18:12.995 pool-15-thread-59 o.a.s.d.n.Nimbus [INFO] Created download session for DemoTest-21-1483568289-stormconf.ser
      2017-01-04 22:18:20.303 timer o.a.s.d.n.Nimbus [INFO] TRANSITION: DemoTest-20-1483567429 REMOVE null false
      2017-01-04 22:18:20.304 timer o.a.s.d.n.Nimbus [ERROR] Error while processing event
      java.lang.RuntimeException: java.lang.NullPointerException
              at org.apache.storm.daemon.nimbus.Nimbus.lambda$delayEvent$16(Nimbus.java:1174)
              at org.apache.storm.StormTimer$StormTimerTask.run(StormTimer.java:83)
      Caused by: java.lang.NullPointerException
              at org.apache.storm.daemon.nimbus.Nimbus.transition(Nimbus.java:1215)
              at org.apache.storm.daemon.nimbus.Nimbus.lambda$delayEvent$16(Nimbus.java:1172)
              ... 1 more
      2017-01-04 22:18:20.304 timer o.a.s.u.Utils [ERROR] Halting process: Error while processing event
      java.lang.RuntimeException: Halting process: Error while processing event
              at org.apache.storm.utils.Utils.exitProcess(Utils.java:1792)
              at org.apache.storm.daemon.nimbus.Nimbus.lambda$new$15(Nimbus.java:1107)
              at org.apache.storm.StormTimer$StormTimerTask.run(StormTimer.java:104)
      2017-01-04 22:18:20.315 Thread-9 o.a.s.d.n.Nimbus [INFO] Shutting down master
      

      The problem is that we are assuming that the base will be non-null which is incorrect leading to NPE.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                raghavgautam Raghav Kumar Gautam
                Reporter:
                raghavgautam Raghav Kumar Gautam
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 40m
                  40m