[HDDS-8980] RocksDB crash in RocksDBStoreMetrics.computeSstFileStat

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 1.4.0
• Labels: pull-request-available
• Target Version/s: 1.4.0
• Assignee: Attila Doroszlai
• Reporter: Attila Doroszlai
• Created: 06/Jul/23 08:31
• Updated: 11/Jul/23 12:25
• Resolved: 11/Jul/23 12:25

Description

Current thread (0x000056142ab93000):  JavaThread "Timer for 'StorageContainerManager' metrics system" daemon [_thread_in_native, id=22190, stack(0x00007fa7691ca000,0x00007fa7692cb000)]

siginfo: si_signo: 11 (SIGSEGV), si_code: 128 (SI_KERNEL), si_addr: 0x0000000000000000

...

C  [librocksdbjni173904417906000101.so+0x2b32ad]  Java_org_rocksdb_RocksDB_getLiveFilesMetaData+0x3d
J 18446  org.rocksdb.RocksDB.getLiveFilesMetaData(J)[Lorg/rocksdb/LiveFileMetaData; (0 bytes) @ 0x00007fa7872b0ee8 [0x00007fa7872b0e40+0xa8]
J 18445 C1 org.rocksdb.RocksDB.getLiveFilesMetaData()Ljava/util/List; (12 bytes) @ 0x00007fa7872b12c4 [0x00007fa7872b1240+0x84]
j  org.apache.hadoop.hdds.utils.db.RocksDatabase.getLiveFilesMetaData()Ljava/util/List;+18
J 18433 C1 org.apache.hadoop.hdds.utils.RocksDBStoreMetrics.computeSstFileStat()Ljava/util/Map; (333 bytes) @ 0x00007fa7872bb804 [0x00007fa7872bb700+0x104]
J 14024 C1 org.apache.hadoop.hdds.utils.RocksDBStoreMetrics.getDBPropertyData(Lorg/apache/hadoop/metrics2/MetricsRecordBuilder;)V (265 bytes) @ 0x00007fa785b6a774 [0x00007fa785b68560+0x2214]
J 18470 C1 org.apache.hadoop.hdds.utils.RocksDBStoreMetrics.getMetrics(Lorg/apache/hadoop/metrics2/MetricsCollector;Z)V (32 bytes) @ 0x00007fa7871bd1dc [0x00007fa7871bc8e0+0x8fc]
J 8866 C2 org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMetrics(Lorg/apache/hadoop/metrics2/impl/MetricsCollectorImpl;Z)Ljava/lang/Iterable; (139 bytes) @ 0x00007fa785e5ece0 [0x00007fa785e5ec60+0x80]
...

• https://github.com/adoroszlai/ozone-build-results/tree/master/2023/07/06/24047/it-filesystem

Issue Links

• is related to: HDDS-6962 Background task for deleting excess snapshot data
• is related to: HDDS-7321 Auto rocksDB small sst files compaction
• is related to: HDDS-7284 JVM crash for rocksdb for read/write after close
• is related to: HDDS-7851 Flush and Compact operation only works on "default" column family
• links to: GitHub Pull Request #5034