[HDDS-10487] Intermittent crash in TestSnapshotDiffManager

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.5.0
• Fix Version/s: 1.5.0
• Component/s: test
• Labels: pull-request-available
• Assignee: Attila Doroszlai
• Reporter: Attila Doroszlai
• Created: 07/Mar/24 15:36
• Updated: 07/Mar/24 20:07
• Resolved: 07/Mar/24 20:07

Description

TestSnapshotDiffManager intermittently crashes:

J 7031  org.rocksdb.RocksDB.put(J[BII[BIIJ)V (0 bytes) @ 0x00007f755d615bbc [0x00007f755d615ac0+0xfc]
J 8461 C2 org.rocksdb.RocksDB.put(Lorg/rocksdb/ColumnFamilyHandle;[B[B)V (21 bytes) @ 0x00007f755dc4157c [0x00007f755dc41520+0x5c]
j  org.apache.hadoop.ozone.om.snapshot.TestSnapshotDiffManager.uploadSnapshotDiffJobToDb(Lorg/apache/hadoop/ozone/om/helpers/SnapshotInfo;Lorg/apache/hadoop/ozone/om/helpers/SnapshotInfo;Lorg/apache/hadoop/ozone/om/helpers/SnapshotDiffJob;)V+79
j  org.apache.hadoop.ozone.om.snapshot.TestSnapshotDiffManager.lambda$testGetSnapshotDiffReportJob$39(Lorg/mockito/invocation/InvocationOnMock;)Ljava/lang/Object;+97
j  org.apache.hadoop.ozone.om.snapshot.TestSnapshotDiffManager$$Lambda$720.answer(Lorg/mockito/invocation/InvocationOnMock;)Ljava/lang/Object;+5
J 8982 C2 org.mockito.internal.creation.bytebuddy.MockMethodInterceptor.doIntercept(Ljava/lang/Object;Ljava/lang/reflect/Method;[Ljava/lang/Object;Lorg/mockito/internal/invocation/RealMethod;Lorg/mockito/invocation/Location;)Ljava/lang/Object; (55 bytes) @ 0x00007f755dfce480 [0x00007f755dfcc280+0x2200]
J 7731 C1 org.mockito.internal.creation.bytebuddy.MockMethodAdvice.handle(Ljava/lang/Object;Ljava/lang/reflect/Method;[Ljava/lang/Object;)Ljava/util/concurrent/Callable; (87 bytes) @ 0x00007f755d6ac53c [0x00007f755d6abd00+0x83c]
j  org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.generateSnapshotDiffReport(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;ZZ)V+203
j  org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.lambda$1(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;ZZ)V+26
j  org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager$$Lambda$497.run()V+36
j  java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V+95
j  java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5

or

j  org.rocksdb.RocksDB.get(J[BIIJ)[B+0
j  org.rocksdb.RocksDB.get(Lorg/rocksdb/ColumnFamilyHandle;[B)[B+13
j  org.apache.hadoop.ozone.om.snapshot.TestSnapshotDiffManager.getSnapshotDiffJobFromDb(Lorg/apache/hadoop/ozone/om/helpers/SnapshotInfo;Lorg/apache/hadoop/ozone/om/helpers/SnapshotInfo;)Lorg/apache/hadoop/ozone/om/helpers/SnapshotDiffJob;+57
j  org.apache.hadoop.ozone.om.snapshot.TestSnapshotDiffManager.lambda$testGetSnapshotDiffReportJob$39(Lorg/mockito/invocation/InvocationOnMock;)Ljava/lang/Object;+34
j  org.apache.hadoop.ozone.om.snapshot.TestSnapshotDiffManager$$Lambda$723.answer(Lorg/mockito/invocation/InvocationOnMock;)Ljava/lang/Object;+5
J 7691 C1 org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(Lorg/mockito/invocation/InvocationOnMock;)Ljava/lang/Object; (68 bytes) @ 0x00007f6b7d6f959c [0x00007f6b7d6f8f20+0x67c]
J 7603 C1 org.mockito.internal.handler.MockHandlerImpl.handle(Lorg/mockito/invocation/Invocation;)Ljava/lang/Object; (306 bytes) @ 0x00007f6b7d4f6b3c [0x00007f6b7d4f4320+0x281c]
J 7602 C1 org.mockito.internal.handler.NullResultGuardian.handle(Lorg/mockito/invocation/Invocation;)Ljava/lang/Object; (39 bytes) @ 0x00007f6b7dac3c3c [0x00007f6b7dac3b20+0x11c]
J 7601 C1 org.mockito.internal.handler.InvocationNotifierHandler.handle(Lorg/mockito/invocation/Invocation;)Ljava/lang/Object; (28 bytes) @ 0x00007f6b7d79e9bc [0x00007f6b7d79e8a0+0x11c]
J 8644 C1 org.mockito.internal.creation.bytebuddy.MockMethodInterceptor.doIntercept(Ljava/lang/Object;Ljava/lang/reflect/Method;[Ljava/lang/Object;Lorg/mockito/internal/invocation/RealMethod;Lorg/mockito/invocation/Location;)Ljava/lang/Object; (55 bytes) @ 0x00007f6b7de2ea8c [0x00007f6b7de2e160+0x92c]
J 7658 C1 org.mockito.internal.creation.bytebuddy.MockMethodAdvice.handle(Ljava/lang/Object;Ljava/lang/reflect/Method;[Ljava/lang/Object;)Ljava/util/concurrent/Callable; (87 bytes) @ 0x00007f6b7d96bebc [0x00007f6b7d96b680+0x83c]
j  org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.generateSnapshotDiffReport(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;ZZ)V+203
j  org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.lambda$1(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;ZZ)V+26
j  org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager$$Lambda$497.run()V+36
j  java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V+95
j  java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5

• https://github.com/adoroszlai/ozone-build-results/blob/master/2024/02/23/29488/unit/hs_err_pid99541.log
• https://github.com/adoroszlai/ozone-build-results/blob/master/2024/02/27/29552/unit/hs_err_pid99470.log
• https://github.com/adoroszlai/ozone-build-results/blob/master/2024/03/05/29746/unit/hs_err_pid100072.log
• https://github.com/adoroszlai/ozone-build-results/blob/master/2024/03/05/29765/unit/hs_err_pid131879.log
• https://github.com/adoroszlai/ozone-build-results/blob/master/2024/03/06/29777/unit/hs_err_pid131875.log

Issue Links

• is related to: HDDS-8315 SnapDiff: Add unit tests
• is related to: HDDS-10149 Bad file descriptor in TestOmSnapshotFsoWithNativeLib.testSnapshotCompactionDag
• links to: GitHub Pull Request #6346