[ZOOKEEPER-4247] NPE while processing message from restarted quorum member

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.6.2
• Fix Version/s: 3.8.0, 3.7.1, 3.6.4
• Component/s: server
• Labels: pull-request-available

Environment

K8S

• Assignee: Mate Szalay-Beko
• Reporter: Devarshi Shah
• Created: 12/Mar/21 08:49
• Updated: 09/Oct/23 11:20
• Resolved: 14/Apr/21 16:56

Description

Problem:

While upgrading K8S cluster, container running Zookeeper (during serving it's client) will rollover one by one.
During this rollover, Null Pointer Exception was observed as below.
After updating to the latest Zookeeper 3.6.2 we still see the problem.
This is happening on a fresh install (and has all the time).

 

Stack-trace:

<from zk-pod-0-log>

2021-02-08T12:42:08.229+0000 [myid:] - ERROR [nioEventLoopGroup-4-1:NettyServerCnxnFactory$CnxnChannelHandler@329] - Unexpected exception in receive
 java.lang.NullPointerException: null
         at org.apache.zookeeper.server.NettyServerCnxn.receiveMessage(NettyServerCnxn.java:518) ~[zookeeper-3.6.2.jar:3.6.2]
         at org.apache.zookeeper.server.NettyServerCnxn.processMessage(NettyServerCnxn.java:368) ~[zookeeper-3.6.2.jar:3.6.2]
         at org.apache.zookeeper.server.NettyServerCnxnFactory$CnxnChannelHandler.channelRead(NettyServerCnxnFactory.java:326) [zookeeper-3.6.2.jar:3.6.2]
         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.50.Final.jar:4.1.50.Final]
         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.50.Final.jar:4.1.50.Final]
         at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.50.Final.jar:4.1.50.Final]
         at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.50.Final.jar:4.1.50.Final]
         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.50.Final.jar:4.1.50.Final]
         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.50.Final.jar:4.1.50.Final]
         at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.50.Final.jar:4.1.50.Final]
         at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.50.Final.jar:4.1.50.Final]
         at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.50.Final.jar:4.1.50.Final]
         at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) [netty-transport-4.1.50.Final.jar:4.1.50.Final]
         at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) [netty-transport-4.1.50.Final.jar:4.1.50.Final]
         at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.50.Final.jar:4.1.50.Final]
         at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.50.Final.jar:4.1.50.Final]
         at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.50.Final.jar:4.1.50.Final]
         at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.50.Final.jar:4.1.50.Final]
         at java.lang.Thread.run(Thread.java:834) [?:?]

 

 

Expectation:

This scenario should be handled and Zookeeper should not print Null Pointer Exception in logs when peer member goes down as a part of the upgrade procedure. 

We are kindly requesting Apache Zookeeper team to fix this issue.

Issue Links

• links to: GitHub Pull Request #1681