[HBASE-25997] NettyRpcFrameDecoder decode request header wrong when handleTooBigRequest

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.4
• Fix Version/s: 3.0.0-alpha-1, 2.3.6, 2.4.5
• Component/s: rpc
• Assignee: Lijin Bin
• Reporter: Lijin Bin
• Created: 11/Jun/21 10:59
• Updated: 23/Dec/22 16:07
• Resolved: 15/Jun/21 06:51

Description

Client write a big request to server, server decode request wrong, so client do not get a RequestTooBigException as expected.

2021-06-11 18:57:27,340 INFO  [RS-EventLoopGroup-1-20] ipc.NettyRpcServer: 
org.apache.hbase.thirdparty.com.google.protobuf.InvalidProtocolBufferException$InvalidWireTypeException: Protocol message tag had invalid wire type.
	at org.apache.hbase.thirdparty.com.google.protobuf.InvalidProtocolBufferException.invalidWireType(InvalidProtocolBufferException.java:111)
	at org.apache.hbase.thirdparty.com.google.protobuf.UnknownFieldSet$Builder.mergeFieldFrom(UnknownFieldSet.java:519)
	at org.apache.hbase.thirdparty.com.google.protobuf.GeneratedMessageV3.parseUnknownField(GeneratedMessageV3.java:298)
	at org.apache.hadoop.hbase.shaded.protobuf.generated.RPCProtos$RequestHeader.<init>(RPCProtos.java:5958)
	at org.apache.hadoop.hbase.shaded.protobuf.generated.RPCProtos$RequestHeader.<init>(RPCProtos.java:5916)
	at org.apache.hadoop.hbase.shaded.protobuf.generated.RPCProtos$RequestHeader$1.parsePartialFrom(RPCProtos.java:7249)
	at org.apache.hadoop.hbase.shaded.protobuf.generated.RPCProtos$RequestHeader$1.parsePartialFrom(RPCProtos.java:7244)
	at org.apache.hadoop.hbase.shaded.protobuf.generated.RPCProtos$RequestHeader$Builder.mergeFrom(RPCProtos.java:6679)
	at org.apache.hadoop.hbase.shaded.protobuf.generated.RPCProtos$RequestHeader$Builder.mergeFrom(RPCProtos.java:6482)
	at org.apache.hbase.thirdparty.com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:420)
	at org.apache.hbase.thirdparty.com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:317)
	at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.mergeFrom(ProtobufUtil.java:2716)
	at org.apache.hadoop.hbase.ipc.NettyRpcFrameDecoder.getHeader(NettyRpcFrameDecoder.java:174)
	at org.apache.hadoop.hbase.ipc.NettyRpcFrameDecoder.handleTooBigRequest(NettyRpcFrameDecoder.java:126)
	at org.apache.hadoop.hbase.ipc.NettyRpcFrameDecoder.decode(NettyRpcFrameDecoder.java:65)
	at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
	at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
	at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:405)
	at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:372)
	at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:355)
	at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:242)
	at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:228)
	at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:221)
	at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1403)
	at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:242)
	at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:228)
	at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:912)
	at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:827)
	at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
	at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
	at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:495)
	at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
	at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:834)

Issue Links

• is duplicated by: HBASE-27544 RegionServer JVM crash when the RPC request size is too big
• is duplicated by: HBASE-26170 handleTooBigRequest in NettyRpcServer didn't skip enough bytes
• links to: GitHub Pull Request #3380