[HBASE-27452] RegionServer JVM crash when scan

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 2.3.5
• Component/s: scan
• Assignee: Unassigned
• Reporter: gaofeng
• Created: 28/Oct/22 10:23
• Updated: 03/Nov/22 03:59

Description

jdk8 + hadoop3 + bucketcache + g1 + hbase2.3.5

In our cluster, we found region servers may be crashed in several cases

hs_err_pid117788.log

#

1. A fatal error has been detected by the Java Runtime Environment:

#

#  SIGSEGV (0xb) at pc=0x00007fdc733a9a4e, pid=117788, tid=0x00007fdc0a34b700

#

1. JRE version: Java(TM) SE Runtime Environment (8.0_171-b11) (build 1.8.0_171-b11)

1. Java VM: Java HotSpot(TM) 64-Bit Server VM (25.171-b11 mixed mode linux-amd64 compressed oops)

1. Problematic frame:

1. j  org.apache.hadoop.hbase.ipc.NettyServerRpcConnection.isConnectionOpen()Z+4

#

1. Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again

#

1. If you would like to submit a bug report, please visit:

1.   http://bugreport.java.com/bugreport/crash.jsp

#

 

---------------  T H R E A D  ---------------

 

Current thread (0x00007fd48004f000):  JavaThread "RpcServer.default.FPBQ.Fifo.handler=17,queue=2,port=16020" daemon [_thread_in_Java, id=118639, stack(0x00007fdc0a24b000,0x00007fdc0a34c000)]

 

regionserver日志

2022-10-28 03:33:03,611 WARN  [RpcServer.default.FPBQ.Fifo.handler=25,queue=0,port=16020] ipc.RpcServer: (responseTooSlow):

{"call":"Scan(org.apache.hadoop.h base.shaded.protobuf.generated.ClientProtos$ScanRequest)","starttimems":"1666899153611","responsesize":"19","method":"Scan","param":"scanner_id: 119667009731 85298703 number_of_rows: 1000 close_scanner: false next_call_seq: 0 client_handles_partials: true client_handles_heartbeats: t <TRUNCATED>","processingtimems ":30000,"client":"10.64.143.22:48550","queuetimems":0,"class":"HRegionServer","scandetails":"table: action_user_track_v2 region: action_user_track_v2,
xE8
x00\\x00\\x00-\\xAA\\xF9\\xFB\\x00\\x00
x01

{\\xCF\\xA6\\xDE\\x09,1645585735023.efb249fd8f4c26d4911d50f14fc30ce8."}

 

2022-10-28 03:33:03,611 WARN  [RpcServer.default.FPBQ.Fifo.handler=17,queue=2,port=16020] hfile.BlockCacheUtil: Caching an already cached block: 58a6823d3958

49099bb226e18acb79b2_SeqId_6__1032126997. This is harmless and can happen in rare cases (see HBASE-8547)

2022-10-28 03:33:03,611 WARN  [RpcServer.default.FPBQ.Fifo.handler=17,queue=2,port=16020] hfile.BlockCacheUtil: Caching an already cached block: eb1c30882099

45f6b5006db45d5fe286_SeqId_4__1032126997. This is harmless and can happen in rare cases (see HBASE-8547)

2022-10-28 03:33:03,612 WARN  [RpcServer.default.FPBQ.Fifo.handler=17,queue=2,port=16020] hfile.BlockCacheUtil: Caching an already cached block: eb1c30882099

45f6b5006db45d5fe286_SeqId_4__1032149468. This is harmless and can happen in rare cases (see HBASE-8547)

Issue Links

• duplicates: HBASE-27455 RegionServer JVM crash when scan