Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-27097

SimpleRpcServer is broken

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • 2.5.0
    • 2.5.0, 3.0.0-alpha-4, 2.4.14
    • rpc
    • None
    • Reviewed

    Description

      Concerns about SimpleRpcServer are not new, and not new to 2.5.  @chenxu noticed a problem on HBASE-23917 back in 2020. After some simple evaluations it seems quite broken.

      When I run an async version of ITLCC against a 2.5.0 cluster configured with hbase.rpc.server.impl=SimpleRpcServer, the client almost immediately stalls because there are too many in flight requests. The logic to pause with too many in flight requests is my own. That's not important. Looking at the server logs it is apparent that SimpleRpcServer is quite broken. Handlers suffer frequent protobuf parse errors and do not properly return responses to the client. This is what stalls my test client. Rather quickly all available request slots are full of requests that will have to time out on the client side.

      Exceptions have three patterns but they all have in common SimpleServerRpcConnection#process. It seems likely the root cause is mismatched expectations or bugs in connection buffer handling in SimpleRpcServer/SimpleServerRpcConnection versus downstream classes that process and parse the buffers. It also seems likely that changes were made to downstream classes like ServerRpcConnection expecting NettyRpcServer's particulars without updating SimpleServerRpcConnection and/or SimpleRpcServer. That said, this is just a superficial analysis.

      1) "Protocol message end-group tag did not match expected tag"

       2022-06-07T16:44:04,625 WARN  [Reader=5,bindAddress=buildbox.localdomain,port=8120] ipc.RpcServer: /127.0.1.1:8120 is unable to read call parameter from client 127.0.0.1
org.apache.hbase.thirdparty.com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.
    at org.apache.hbase.thirdparty.com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:129) ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
    at org.apache.hbase.thirdparty.com.google.protobuf.CodedInputStream$ByteInputDecoder.checkLastTagWas(CodedInputStream.java:4034) ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
    at org.apache.hbase.thirdparty.com.google.protobuf.CodedInputStream$ByteInputDecoder.readMessage(CodedInputStream.java:4275) ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
    at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue.<init>(ClientProtos.java:10520) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
    at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue.<init>(ClientProtos.java:10464) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
    at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue$1.parsePartialFrom(ClientProtos.java:12251) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
    at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue$1.parsePartialFrom(ClientProtos.java:12245) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
    at org.apache.hbase.thirdparty.com.google.protobuf.CodedInputStream$ByteInputDecoder.readMessage(CodedInputStream.java:4274) ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
    at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto.<init>(ClientProtos.java:9981) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
    at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto.<init>(ClientProtos.java:9910) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
    at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$1.parsePartialFrom(ClientProtos.java:14097) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
    at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$1.parsePartialFrom(ClientProtos.java:14091) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
    at org.apache.hbase.thirdparty.com.google.protobuf.CodedInputStream$ByteInputDecoder.readMessage(CodedInputStream.java:4274) ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
    at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutateRequest.<init>(ClientProtos.java:14251) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
    at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutateRequest.<init>(ClientProtos.java:14190) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
    at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutateRequest$1.parsePartialFrom(ClientProtos.java:15304) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
    at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutateRequest$1.parsePartialFrom(ClientProtos.java:15298) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
    at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutateRequest$Builder.mergeFrom(ClientProtos.java:14860) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
    at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutateRequest$Builder.mergeFrom(ClientProtos.java:14651) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
    at org.apache.hbase.thirdparty.com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:420) ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
    at org.apache.hbase.thirdparty.com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:317) ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
    at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.mergeFrom(ProtobufUtil.java:2638) ~[hbase-client-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
    at org.apache.hadoop.hbase.ipc.ServerRpcConnection.processRequest(ServerRpcConnection.java:644) ~[hbase-server-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
    at org.apache.hadoop.hbase.ipc.ServerRpcConnection.processOneRpc(ServerRpcConnection.java:444) ~[hbase-server-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
    at org.apache.hadoop.hbase.ipc.SimpleServerRpcConnection.process(SimpleServerRpcConnection.java:285) ~[hbase-server-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
    at org.apache.hadoop.hbase.ipc.SimpleServerRpcConnection.readAndProcess(SimpleServerRpcConnection.java:251) ~[hbase-server-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
    at org.apache.hadoop.hbase.ipc.SimpleRpcServer$Listener.doRead(SimpleRpcServer.java:318) ~[hbase-server-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
    at org.apache.hadoop.hbase.ipc.SimpleRpcServer$Listener$Reader.doRunLoop(SimpleRpcServer.java:180) ~[hbase-server-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
    at org.apache.hadoop.hbase.ipc.SimpleRpcServer$Listener$Reader.run(SimpleRpcServer.java:153) ~[hbase-server-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]

      2) "Protocol message tag had invalid wire type."

      2022-06-07T16:44:04,705 WARN  [Reader=6,bindAddress=buildbox.localdomain,port=8120] ipc.RpcServer: /127.0.1.1:8120 is unable to read call parameter from client 127.0.0.1
org.apache.hbase.thirdparty.com.google.protobuf.InvalidProtocolBufferException$InvalidWireTypeException: Protocol message tag had invalid wire type.
	at org.apache.hbase.thirdparty.com.google.protobuf.InvalidProtocolBufferException.invalidWireType(InvalidProtocolBufferException.java:134) ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
	at org.apache.hbase.thirdparty.com.google.protobuf.UnknownFieldSet$Builder.mergeFieldFrom(UnknownFieldSet.java:527) ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
	at org.apache.hbase.thirdparty.com.google.protobuf.GeneratedMessageV3.parseUnknownField(GeneratedMessageV3.java:320) ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
	at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue$QualifierValue.<init>(ClientProtos.java:10700) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
	at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue$QualifierValue.<init>(ClientProtos.java:10620) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
	at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue$QualifierValue$1.parsePartialFrom(ClientProtos.java:11481) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
	at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue$QualifierValue$1.parsePartialFrom(ClientProtos.java:11475) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
	at org.apache.hbase.thirdparty.com.google.protobuf.CodedInputStream$ByteInputDecoder.readMessage(CodedInputStream.java:4274) ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
	at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue.<init>(ClientProtos.java:10520) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
	at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue.<init>(ClientProtos.java:10464) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
	at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue$1.parsePartialFrom(ClientProtos.java:12251) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
	at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue$1.parsePartialFrom(ClientProtos.java:12245) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
	at org.apache.hbase.thirdparty.com.google.protobuf.CodedInputStream$ByteInputDecoder.readMessage(CodedInputStream.java:4274) ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
	at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto.<init>(ClientProtos.java:9981) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
	at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto.<init>(ClientProtos.java:9910) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
	at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$1.parsePartialFrom(ClientProtos.java:14097) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
	at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$1.parsePartialFrom(ClientProtos.java:14091) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
	at org.apache.hbase.thirdparty.com.google.protobuf.CodedInputStream$ByteInputDecoder.readMessage(CodedInputStream.java:4274) ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
	at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutateRequest.<init>(ClientProtos.java:14251) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
	at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutateRequest.<init>(ClientProtos.java:14190) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
	at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutateRequest$1.parsePartialFrom(ClientProtos.java:15304) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
	at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutateRequest$1.parsePartialFrom(ClientProtos.java:15298) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
	at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutateRequest$Builder.mergeFrom(ClientProtos.java:14860) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
	at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutateRequest$Builder.mergeFrom(ClientProtos.java:14651) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
	at org.apache.hbase.thirdparty.com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:420) ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
	at org.apache.hbase.thirdparty.com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:317) ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
	at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.mergeFrom(ProtobufUtil.java:2638) ~[hbase-client-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
	at org.apache.hadoop.hbase.ipc.ServerRpcConnection.processRequest(ServerRpcConnection.java:644) ~[hbase-server-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
	at org.apache.hadoop.hbase.ipc.ServerRpcConnection.processOneRpc(ServerRpcConnection.java:444) ~[hbase-server-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
	at org.apache.hadoop.hbase.ipc.SimpleServerRpcConnection.process(SimpleServerRpcConnection.java:285) ~[hbase-server-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
	at org.apache.hadoop.hbase.ipc.SimpleServerRpcConnection.readAndProcess(SimpleServerRpcConnection.java:251) ~[hbase-server-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
	at org.apache.hadoop.hbase.ipc.SimpleRpcServer$Listener.doRead(SimpleRpcServer.java:318) ~[hbase-server-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
	at org.apache.hadoop.hbase.ipc.SimpleRpcServer$Listener$Reader.doRunLoop(SimpleRpcServer.java:180) ~[hbase-server-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
	at org.apache.hadoop.hbase.ipc.SimpleRpcServer$Listener$Reader.run(SimpleRpcServer.java:153) ~[hbase-server-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:829) ~[?:?]

      3) "While parsing a protocol message, the input ended unexpectedly in the middle of a field."

      2022-06-07T16:44:04,885 WARN  [Reader=9,bindAddress=buildbox.localdomain,port=8120] ipc.RpcServer: /127.0.1.1:8120 is unable to read call parameter from client 127.0.0.1
org.apache.hbase.thirdparty.com.google.protobuf.InvalidProtocolBufferException: While parsing a protocol message, the input ended unexpectedly in the middle of a field.  This could mean either that the input has been truncated or that an embedded message misreported its own length.
	at org.apache.hbase.thirdparty.com.google.protobuf.InvalidProtocolBufferException.truncatedMessage(InvalidProtocolBufferException.java:107) ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
	at org.apache.hbase.thirdparty.com.google.protobuf.CodedInputStream$ByteInputDecoder.readRawLittleEndian64(CodedInputStream.java:4478) ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
	at org.apache.hbase.thirdparty.com.google.protobuf.CodedInputStream$ByteInputDecoder.readFixed64(CodedInputStream.java:4167) ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
	at org.apache.hbase.thirdparty.com.google.protobuf.UnknownFieldSet$Builder.mergeFieldFrom(UnknownFieldSet.java:511) ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
	at org.apache.hbase.thirdparty.com.google.protobuf.GeneratedMessageV3.parseUnknownField(GeneratedMessageV3.java:320) ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
	at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue$QualifierValue.<init>(ClientProtos.java:10700) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
	at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue$QualifierValue.<init>(ClientProtos.java:10620) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
	at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue$QualifierValue$1.parsePartialFrom(ClientProtos.java:11481) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
	at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue$QualifierValue$1.parsePartialFrom(ClientProtos.java:11475) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
	at org.apache.hbase.thirdparty.com.google.protobuf.CodedInputStream$ByteInputDecoder.readMessage(CodedInputStream.java:4274) ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
	at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue.<init>(ClientProtos.java:10520) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
	at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue.<init>(ClientProtos.java:10464) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
	at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue$1.parsePartialFrom(ClientProtos.java:12251) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
	at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue$1.parsePartialFrom(ClientProtos.java:12245) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
	at org.apache.hbase.thirdparty.com.google.protobuf.CodedInputStream$ByteInputDecoder.readMessage(CodedInputStream.java:4274) ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
	at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto.<init>(ClientProtos.java:9981) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
	at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto.<init>(ClientProtos.java:9910) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
	at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$1.parsePartialFrom(ClientProtos.java:14097) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
	at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$1.parsePartialFrom(ClientProtos.java:14091) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
	at org.apache.hbase.thirdparty.com.google.protobuf.CodedInputStream$ByteInputDecoder.readMessage(CodedInputStream.java:4274) ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
	at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutateRequest.<init>(ClientProtos.java:14251) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
	at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutateRequest.<init>(ClientProtos.java:14190) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
	at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutateRequest$1.parsePartialFrom(ClientProtos.java:15304) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
	at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutateRequest$1.parsePartialFrom(ClientProtos.java:15298) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
	at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutateRequest$Builder.mergeFrom(ClientProtos.java:14860) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
	at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutateRequest$Builder.mergeFrom(ClientProtos.java:14651) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
	at org.apache.hbase.thirdparty.com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:420) ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
	at org.apache.hbase.thirdparty.com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:317) ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
	at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.mergeFrom(ProtobufUtil.java:2638) ~[hbase-client-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
	at org.apache.hadoop.hbase.ipc.ServerRpcConnection.processRequest(ServerRpcConnection.java:644) ~[hbase-server-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
	at org.apache.hadoop.hbase.ipc.ServerRpcConnection.processOneRpc(ServerRpcConnection.java:444) ~[hbase-server-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
	at org.apache.hadoop.hbase.ipc.SimpleServerRpcConnection.process(SimpleServerRpcConnection.java:285) ~[hbase-server-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
	at org.apache.hadoop.hbase.ipc.SimpleServerRpcConnection.readAndProcess(SimpleServerRpcConnection.java:251) ~[hbase-server-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
	at org.apache.hadoop.hbase.ipc.SimpleRpcServer$Listener.doRead(SimpleRpcServer.java:318) ~[hbase-server-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
	at org.apache.hadoop.hbase.ipc.SimpleRpcServer$Listener$Reader.doRunLoop(SimpleRpcServer.java:180) ~[hbase-server-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
	at org.apache.hadoop.hbase.ipc.SimpleRpcServer$Listener$Reader.run(SimpleRpcServer.java:153) ~[hbase-server-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:829) ~[?:?]

      Attachments

        1. MultiByteBuff.patch
          0.6 kB
          Lijin Bin

        Issue Links

          is blocked by

          Bug - A problem which impairs or prevents the functions of the product. HBASE-27170 ByteBuffAllocator leak when decompressing blocks near minSizeForReservoirUse

          • Major - Major loss of function.
          • Resolved
          relates to

          Bug - A problem which impairs or prevents the functions of the product. HBASE-26708 Netty "leak detected" and OutOfDirectMemoryError due to direct memory buffering with SASL implementation

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved
          links to

          Web Link GitHub Pull Request #4520

          Web Link GitHub Pull Request #4521

          Web Link GitHub Pull Request #4613

          Activity

            People

              apurtell Andrew Kyle Purtell
              apurtell Andrew Kyle Purtell
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: