[HDDS-6763] SCM crashed while getting certificate from SCMCertStore

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Assignee: Unassigned
• Reporter: Neil Joshi
• Created: 17/May/22 21:04
• Updated: 30/May/23 18:27
• Resolved: 30/May/23 18:27

Description

With secure HA cluster, scm crashes in safe-mode while retrieving certificate from SCMCertStore.

 

 

Current thread (0x00007fda6c017800):  JavaThread "Listener at 0.0.0.0/9860" [_thread_in_native, id=171, stack(0x00007fda7399b000,0x00007fda73a9c000)]
Stack: [0x00007fda7399b000,0x00007fda73a9c000],  sp=0x00007fda73a99508,  free space=1017k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  0x00007fda6d2426a0
C  [librocksdbjni6738631227401849863.so+0x28d8a2]  Java_org_rocksdb_RocksDB_get__J_3BIIJ+0x62
j  org.rocksdb.RocksDB.get(J[BIIJ)[B+0
j  org.rocksdb.RocksDB.get(Lorg/rocksdb/ColumnFamilyHandle;[B)[B+13
j  org.apache.hadoop.hdds.utils.db.RDBTable.get([B)[B+9
j  org.apache.hadoop.hdds.utils.db.RDBTable.get(Ljava/lang/Object;)Ljava/lang/Object;+5
j  org.apache.hadoop.hdds.utils.db.TypedTable.getFromTable(Ljava/lang/Object;)Ljava/lang/Object;+14
j  org.apache.hadoop.hdds.utils.db.TypedTable.get(Ljava/lang/Object;)Ljava/lang/Object;+61
j  org.apache.hadoop.hdds.scm.server.SCMCertStore.getCertificateByID(Ljava/math/BigInteger;Lorg/apache/hadoop/hdds/security/x509/certificate/authority/CertificateStore$CertType;)Ljava/security/cert/X509Ce
rtificate;+17
v  ~StubRoutines::call_stub
V  [libjvm.so+0x8ce625]  JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, Thread*)+0x395
V  [libjvm.so+0xced613]  invoke(InstanceKlass*, methodHandle const&, Handle, bool, objArrayHandle, BasicType, objArrayHandle, bool, Thread*) [clone .isra.199]+0x603
V  [libjvm.so+0xcee367]  Reflection::invoke_method(oopDesc*, Handle, objArrayHandle, Thread*)+0x137
V  [libjvm.so+0x97d953]  JVM_InvokeMethod+0x1a3
j  jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Ljava/lang/reflect/Method;Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;+0 java.base@11.0.14.1
j  jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;+100 java.base@11.0.14.1
J 3798 c1 jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; java.base@11.0.14.1 (10 bytes) @ 0x00007fda5556b674 [0x00007fda5556b540+0x000000
0000000134]
J 3797 c1 java.lang.reflect.Method.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; java.base@11.0.14.1 (65 bytes) @ 0x00007fda5556b10c [0x00007fda5556ada0+0x000000000000036c]
j  org.apache.hadoop.hdds.scm.ha.SCMHAInvocationHandler.invokeLocal(Ljava/lang/reflect/Method;[Ljava/lang/Object;)Ljava/lang/Object;+35
j  org.apache.hadoop.hdds.scm.ha.SCMHAInvocationHandler.invoke(Ljava/lang/Object;Ljava/lang/reflect/Method;[Ljava/lang/Object;)Ljava/lang/Object;+33
j  com.sun.proxy.$Proxy21.getCertificateByID(Ljava/math/BigInteger;Lorg/apache/hadoop/hdds/security/x509/certificate/authority/CertificateStore$CertType;)Ljava/security/cert/X509Certificate;+20
j  org.apache.hadoop.hdds.scm.server.StorageContainerManager.persistSCMCertificates()V+76
j  org.apache.hadoop.hdds.scm.server.StorageContainerManager.start()V+155
j  org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter$SCMStarterHelper.start(Lorg/apache/hadoop/hdds/conf/OzoneConfiguration;)V+6
j  org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter.startScm()V+8

hs_err_pid8.log

 

Issue Links

• is related to: HDDS-6732 Follower SCM crashed during snapshot installation
• is related to: HDDS-6685 Follower OM crashed when validating S3 auth info.
• is related to: HDDS-6740 SCM crashed while taking snapshot