[HDDS-6114] Intermittent error due to Failed to init RocksDB

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.3.0
• Component/s: test
• Labels: pull-request-available
• Target Version/s: 1.3.0
• Assignee: Attila Doroszlai
• Reporter: Attila Doroszlai
• Created: 16/Dec/21 19:24
• Updated: 12/Jan/22 19:28
• Resolved: 12/Jan/22 19:28

Description

https://github.com/elek/ozone-build-results/blob/master/2021/11/23/11642/it-ozone/hadoop-ozone/integration-test/org.apache.hadoop.ozone.scm.TestStorageContainerManagerHA.txt

Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 176.373 s <<< FAILURE! - in org.apache.hadoop.ozone.scm.TestStorageContainerManagerHA
testBootStrapSCM  Time elapsed: 23.788 s  <<< ERROR!
java.io.IOException: Failed init RocksDB, db path : /home/runner/work/ozone/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-7ab73b5c-8ac7-4e67-b2fd-a2c0bde55e77/omNode-2/om.db, exception :org.rocksdb.RocksDBException lock hold by current process, acquire time 1637654016 acquiring thread 139638819440384: /home/runner/work/ozone/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-7ab73b5c-8ac7-4e67-b2fd-a2c0bde55e77/omNode-2/om.db/LOCK: No locks available; status : IOError; message : lock hold by current process, acquire time 1637654016 acquiring thread 139638819440384: /home/runner/work/ozone/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-7ab73b5c-8ac7-4e67-b2fd-a2c0bde55e77/omNode-2/om.db/LOCK: No locks available
	at org.apache.hadoop.hdds.utils.HddsServerUtil.toIOException(HddsServerUtil.java:564)
	at org.apache.hadoop.hdds.utils.db.RDBStore.<init>(RDBStore.java:164)
	at org.apache.hadoop.hdds.utils.db.DBStoreBuilder.build(DBStoreBuilder.java:191)
	at org.apache.hadoop.ozone.om.OmMetadataManagerImpl.loadDB(OmMetadataManagerImpl.java:397)
	at org.apache.hadoop.ozone.om.OmMetadataManagerImpl.loadDB(OmMetadataManagerImpl.java:387)
	at org.apache.hadoop.ozone.om.OmMetadataManagerImpl.start(OmMetadataManagerImpl.java:379)
	at org.apache.hadoop.ozone.om.OmMetadataManagerImpl.<init>(OmMetadataManagerImpl.java:246)
	at org.apache.hadoop.ozone.om.OzoneManager.instantiateServices(OzoneManager.java:640)
	at org.apache.hadoop.ozone.om.OzoneManager.<init>(OzoneManager.java:528)
	at org.apache.hadoop.ozone.om.OzoneManager.createOm(OzoneManager.java:606)
	at org.apache.hadoop.ozone.MiniOzoneHAClusterImpl$Builder.createOMService(MiniOzoneHAClusterImpl.java:494)
	at org.apache.hadoop.ozone.MiniOzoneHAClusterImpl$Builder.build(MiniOzoneHAClusterImpl.java:405)
	at org.apache.hadoop.ozone.scm.TestStorageContainerManagerHA.init(TestStorageContainerManagerHA.java:103)

Issue Links

• is a child of: HDDS-5626 Track and Address Flaky tests
• links to: GitHub Pull Request #2947