[STORM-3764] Nimbus has internal scheduling errors when backtracking

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.6.0
• Component/s: storm-server
• Assignee: Bipin Prasad
• Reporter: Bipin Prasad
• Created: 02/Apr/21 19:13
• Updated: 25/Oct/23 08:43
• Resolved: 25/Oct/23 08:43

Description

While backtracking in scheduling code, Ackers are skipped. However, under certain circumstances a null WorkerSlot is passed as parameter to SchedulingSearcherState.backtrack(). This null key is used to retrieve an entry from Map. The returned null map value causes NPE.

2021-03-30 15:27:03.212 o.a.s.s.r.ResourceAwareScheduler timer [ERROR] mx3_test-5-1614799784 Internal Error - Exception thrown when scheduling. Please check logs for details java.util.concurrent.ExecutionException: java.lang.NullPointerException
    at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_262]
    at java.util.concurrent.FutureTask.get(FutureTask.java:206) ~[?:1.8.0_262]
    at org.apache.storm.scheduler.resource.ResourceAwareScheduler.scheduleTopology(ResourceAwareScheduler.java:191) ~[storm-server-2.3.0.y.jar:2.3.0.y]
    at org.apache.storm.scheduler.resource.ResourceAwareScheduler.schedule(ResourceAwareScheduler.java:132) ~[storm-server-2.3.0.y.jar:2.3.0.y]
    at org.apache.storm.scheduler.blacklist.BlacklistScheduler.schedule(BlacklistScheduler.java:131) ~[storm-server-2.3.0.y.jar:2.3.0.y]
    at org.apache.storm.daemon.nimbus.Nimbus.computeNewSchedulerAssignments(Nimbus.java:2314) ~[storm-server-2.3.0.y.jar:2.3.0.y]
    at org.apache.storm.daemon.nimbus.Nimbus.lambda$lockingMkAssignments$44(Nimbus.java:2493) ~[storm-server-2.3.0.y.jar:2.3.0.y]
    at org.apache.storm.daemon.nimbus.Nimbus.executeWithLock(Nimbus.java:4549) ~[storm-server-2.3.0.y.jar:2.3.0.y]
    at org.apache.storm.daemon.nimbus.Nimbus.lockingMkAssignments(Nimbus.java:2491) ~[storm-server-2.3.0.y.jar:2.3.0.y]
    at org.apache.storm.daemon.nimbus.Nimbus.mkAssignments(Nimbus.java:2478) ~[storm-server-2.3.0.y.jar:2.3.0.y]
    at org.apache.storm.daemon.nimbus.Nimbus.mkAssignments(Nimbus.java:2415) ~[storm-server-2.3.0.y.jar:2.3.0.y]
    at org.apache.storm.daemon.nimbus.Nimbus.lambda$launchServer$17(Nimbus.java:1420) ~[storm-server-2.3.0.y.jar:2.3.0.y]
    at org.apache.storm.StormTimer$1.run(StormTimer.java:110) [storm-client-2.3.0.y.jar:2.3.0.y]
    at org.apache.storm.StormTimer$StormTimerTask.run(StormTimer.java:226) [storm-client-2.3.0.y.jar:2.3.0.y]

Caused by: java.lang.NullPointerException
    at org.apache.storm.scheduler.resource.strategies.scheduling.SchedulingSearcherState.backtrack(SchedulingSearcherState.java:292) ~[storm-server-2.3.0.y.jar:2.3.0.y]
    at org.apache.storm.scheduler.resource.strategies.scheduling.BaseResourceAwareStrategy.scheduleExecutorsOnNodes(BaseResourceAwareStrategy.java:559) ~[storm-server-2.3.0.y.jar:2.3.0.y]
    at org.apache.storm.scheduler.resource.strategies.scheduling.BaseResourceAwareStrategy.schedule(BaseResourceAwareStrategy.java:172) ~[storm-server-2.3.0.y.jar:2.3.0.y]
    at org.apache.storm.scheduler.resource.ResourceAwareScheduler.lambda$scheduleTopology$1(ResourceAwareScheduler.java:189) ~[storm-server-2.3.0.y.jar:2.3.0.y]     
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_262]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_262]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_262]
    at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_262]