[STORM-3751] NPE in WorkerState.transferLocalBatch

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.2.0
• Component/s: storm-client
• Assignee: Unassigned
• Reporter: Arwin S Tio
• Created: 07/Mar/21 06:20
• Updated: 21/Jul/22 09:01

Description

Hello,

 

I've recently upgraded to Storm 2.2.0 and have been getting this error:

 

2021-03-07 04:36:51.061 o.a.s.m.n.StormServerHandler Netty-server-localhost-6700-worker-1 [ERROR] server errors in handling the request
java.lang.NullPointerException: null
        at org.apache.storm.daemon.worker.WorkerState.transferLocalBatch(WorkerState.java:543) ~[storm-client-2.2.0.jar:2.2.0]
        at org.apache.storm.messaging.DeserializingConnectionCallback.recv(DeserializingConnectionCallback.java:71) ~[storm-client-2.2.0.jar:2.2.0]
        at org.apache.storm.messaging.netty.Server.enqueue(Server.java:146) ~[storm-client-2.2.0.jar:2.2.0]
        at org.apache.storm.messaging.netty.Server.received(Server.java:264) ~[storm-client-2.2.0.jar:2.2.0]
        at org.apache.storm.messaging.netty.StormServerHandler.channelRead(StormServerHandler.java:51) ~[storm-client-2.2.0.jar:2.2.0]
        at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [storm-shaded-deps-2.2.0.jar:2.2.0]
        at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [storm-shaded-deps-2.2.0.jar:2.2.0]
        at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [storm-shaded-deps-2.2.0.jar:2.2.0]
        at org.apache.storm.shade.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) [storm-shaded-deps-2.2.0.jar:2.2.0]
        at org.apache.storm.shade.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297) [storm-shaded-deps-2.2.0.jar:2.2.0]
        at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [storm-shaded-deps-2.2.0.jar:2.2.0]
        at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [storm-shaded-deps-2.2.0.jar:2.2.0]
        at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [storm-shaded-deps-2.2.0.jar:2.2.0]
        at org.apache.storm.shade.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [storm-shaded-deps-2.2.0.jar:2.2.0]
        at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [storm-shaded-deps-2.2.0.jar:2.2.0]
        at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [storm-shaded-deps-2.2.0.jar:2.2.0]
        at org.apache.storm.shade.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [storm-shaded-deps-2.2.0.jar:2.2.0]
        at org.apache.storm.shade.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [storm-shaded-deps-2.2.0.jar:2.2.0]
        at org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [storm-shaded-deps-2.2.0.jar:2.2.0]
        at org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579) [storm-shaded-deps-2.2.0.jar:2.2.0]
        at org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496) [storm-shaded-deps-2.2.0.jar:2.2.0]
        at org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [storm-shaded-deps-2.2.0.jar:2.2.0]
        at org.apache.storm.shade.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) [storm-shaded-deps-2.2.0.jar:2.2.0]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_272]
2021-03-07 04:36:51.061 o.a.s.m.n.StormServerHandler Netty-server-localhost-6700-worker-1 [INFO] Received error in netty thread.. terminating server... 

 
This issue happens every 20-30 minutes and causes the workers to die/restart.

It seems related to https://issues.apache.org/jira/browse/STORM-3141 but seems to have been fixed in 2.0. 

I am happy to provide more information but at the moment am unsure of what is relevant.

I have a suspicion that this is related to load-aware localOrShuffleGrouping ("LoadAwareShuffleGrouping") because this issue seems to have started when I switched the Grouping, but again, not sure if it's actually related.