[HDDS-10149] Bad file descriptor in TestOmSnapshotFsoWithNativeLib.testSnapshotCompactionDag

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Component/s: Snapshot
• Labels: pull-request-available
• Target Version/s: 1.4.1
• Assignee: Swaminathan Balachandran
• Reporter: Attila Doroszlai
• Created: 17/Jan/24 14:22
• Updated: 07/Mar/24 16:37
• Resolved: 27/Feb/24 21:23

Description

TestOmSnapshotFsoWithNativeLib failed in a CI run where it was really using the native library:

org.apache.hadoop.ozone.om.snapshot.TestOmSnapshotFsoWithNativeLib.testSnapshotCompactionDag -- Time elapsed: 2.844 s <<< ERROR!
java.io.IOException: RocksDatabase[/home/runner/work/ozone/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-d6a9ee09-9b79-4e8b-9241-4df752521eb9/ozone-meta/om.db]: Failed to compactRange; status : IOError; message : While pread offset 0 len 176: /home/runner/work/ozone/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-d6a9ee09-9b79-4e8b-9241-4df752521eb9/ozone-meta/om.db/000508.sst: Bad file descriptor
	at org.apache.hadoop.hdds.utils.HddsServerUtil.toIOException(HddsServerUtil.java:667)
	at org.apache.hadoop.hdds.utils.db.RocksDatabase.toIOException(RocksDatabase.java:98)
	at org.apache.hadoop.hdds.utils.db.RocksDatabase.compactRange(RocksDatabase.java:595)
	at org.apache.hadoop.hdds.utils.db.RocksDatabase.compactDB(RocksDatabase.java:551)
	at org.apache.hadoop.hdds.utils.db.RDBStore.compactDB(RDBStore.java:216)
	at org.apache.hadoop.ozone.om.snapshot.TestOmSnapshot.testSnapshotCompactionDag(TestOmSnapshot.java:2415)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at java.util.ArrayList.forEach(ArrayList.java:1259)
	at java.util.ArrayList.forEach(ArrayList.java:1259)
Caused by: org.rocksdb.RocksDBException: While pread offset 0 len 176: /home/runner/work/ozone/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-d6a9ee09-9b79-4e8b-9241-4df752521eb9/ozone-meta/om.db/000508.sst: Bad file descriptor
	at org.rocksdb.RocksDB.compactRange(Native Method)
	at org.rocksdb.RocksDB.compactRange(RocksDB.java:3416)
	at org.apache.hadoop.hdds.utils.db.RocksDatabase.compactRange(RocksDatabase.java:592)
	... 6 more

CC smeng, hemantk

Issue Links

• Discovered while testing: HDDS-10148 TestOmSnapshotFsoWithNativeLib should be tagged as native test
• is duplicated by: HDDS-9433 [snapshot] OM shutsdown on RocksDB failure when performing distcp of snapshots
• is duplicated by: HDDS-9711 [snapshot] RuntimeException encountered when generating snapshotDiff report between 2 snapshots
• relates to: HDDS-10487 Intermittent crash in TestSnapshotDiffManager
• links to: GitHub Pull Request #6182