[HDDS-6740] SCM crashed while taking snapshot

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.4.0
• Component/s: SCM
• Target Version/s: 1.4.0
• Assignee: Unassigned
• Reporter: Attila Doroszlai
• Created: 12/May/22 13:34
• Updated: 08/Sep/23 08:06
• Resolved: 08/Sep/23 08:06

Description

https://github.com/adoroszlai/ozone-build-results/blob/6ef0d20d9892d72c664a3d9671c22738ad2d6ce0/2022/05/12/14985/it-flaky/output.log#L101-L102

[INFO] Running org.apache.hadoop.hdds.scm.TestSCMInstallSnapshot
[ERROR] Aborted (core dumped)

https://github.com/adoroszlai/ozone-build-results/blob/6ef0d20d9892d72c664a3d9671c22738ad2d6ce0/2022/05/12/14985/it-flaky/hs_err_pid16149.log#L94-L110

Stack: [0x00007ff97ee05000,0x00007ff97ef06000],  sp=0x00007ff97ef04208,  free space=1020k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [librocksdbjni3641929238250681197.so+0x304358]  rocksdb::GetColumnFamilyID(rocksdb::ColumnFamilyHandle*)+0x8
C  [librocksdbjni3641929238250681197.so+0x4b9bfa]  rocksdb::WriteBatch::Put(rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, rocksdb::Slice const&)+0x2a
C  [librocksdbjni3641929238250681197.so+0x2bbeeb]
C  [librocksdbjni3641929238250681197.so+0x2bcd07]  Java_org_rocksdb_WriteBatch_put__J_3BI_3BIJ+0xe7
j  org.rocksdb.WriteBatch.put(J[BI[BIJ)V+0
j  org.rocksdb.AbstractWriteBatch.put(Lorg/rocksdb/ColumnFamilyHandle;[B[B)V+15
j  org.apache.hadoop.hdds.utils.db.RDBBatchOperation.put(Lorg/rocksdb/ColumnFamilyHandle;[B[B)V+12
j  org.apache.hadoop.hdds.utils.db.RDBTable.putWithBatch(Lorg/apache/hadoop/hdds/utils/db/BatchOperation;[B[B)V+28
j  org.apache.hadoop.hdds.utils.db.RDBTable.putWithBatch(Lorg/apache/hadoop/hdds/utils/db/BatchOperation;Ljava/lang/Object;Ljava/lang/Object;)V+15
j  org.apache.hadoop.hdds.utils.db.TypedTable.putWithBatch(Lorg/apache/hadoop/hdds/utils/db/BatchOperation;Ljava/lang/Object;Ljava/lang/Object;)V+46
j  org.apache.hadoop.hdds.scm.ha.SCMHADBTransactionBufferImpl.flush()V+35
j  org.apache.hadoop.hdds.scm.ha.SCMStateMachine.takeSnapshot()J+140
j  org.apache.ratis.server.impl.StateMachineUpdater.takeSnapshot()V+15
j  org.apache.ratis.server.impl.StateMachineUpdater.checkAndTakeSnapshot(Lorg/apache/ratis/util/MemoizedSupplier;)V+29
j  org.apache.ratis.server.impl.StateMachineUpdater.run()V+35

from: https://github.com/apache/ozone/runs/6401745544

Issue Links

• is related to: HDDS-6732 Follower SCM crashed during snapshot installation
• relates to: HDDS-6763 SCM crashed while getting certificate from SCMCertStore