[HDDS-9486] Deadlock between RocksDBCheckpointDiffer#pruneSstFiles and OMDBCheckpointServlet#getCheckpoint. Also causing intermittent fork timeout in TestSnapshotBackgroundServices.

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.4.0
• Fix Version/s: 2.0.0, 1.4.1
• Component/s: Snapshot
• Labels: pull-request-available
• Assignee: Hemant Kumar
• Reporter: Attila Doroszlai
• Created: 18/Oct/23 09:49
• Updated: 13/Aug/24 18:00
• Resolved: 27/Jan/24 13:58

Description

Surefire fork for TestSnapshotBackgroundServices intermittently times out.

• https://github.com/adoroszlai/ozone-build-results/blob/master/2023/09/07/25178/it-om/output.log
• https://github.com/adoroszlai/ozone-build-results/blob/master/2023/09/14/25374/it-om/output.log

CC hemantk, mladjangadzic

"CompactionDagPruningService" 
   java.lang.Thread.State: WAITING
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
        at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
        at org.apache.hadoop.ozone.lock.BootstrapStateHandler$Lock.lock(BootstrapStateHandler.java:31)
        at org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.pruneSstFiles(RocksDBCheckpointDiffer.java:1506)
        at org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer$$Lambda$573/124020389.run(Unknown Source)

"qtp555959536-13964" 
   java.lang.Thread.State: BLOCKED
        at org.apache.hadoop.ozone.om.OMDBCheckpointServlet.getCheckpoint(OMDBCheckpointServlet.java:255)
        at org.apache.hadoop.hdds.utils.DBCheckpointServlet.generateSnapshotCheckpoint(DBCheckpointServlet.java:200)
        at org.apache.hadoop.hdds.utils.DBCheckpointServlet.doPost(DBCheckpointServlet.java:321)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:523)

 "main" 
   java.lang.Thread.State: BLOCKED
        at org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.close(RocksDBCheckpointDiffer.java:340)
        at org.apache.hadoop.hdds.utils.IOUtils.close(IOUtils.java:78)
        at org.apache.hadoop.hdds.utils.IOUtils.close(IOUtils.java:64)
        at org.apache.hadoop.hdds.utils.IOUtils.closeQuietly(IOUtils.java:92)
        at org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer$RocksDBCheckpointDifferHolder.invalidateCacheEntry(RocksDBCheckpointDiffer.java:1591)
        at org.apache.hadoop.hdds.utils.db.RDBStore.close(RDBStore.java:224)
        at org.apache.hadoop.ozone.om.OmMetadataManagerImpl.stop(OmMetadataManagerImpl.java:753)
        at org.apache.hadoop.ozone.om.OzoneManager.stop(OzoneManager.java:2246)
        at org.apache.hadoop.ozone.MiniOzoneHAClusterImpl.stop(MiniOzoneHAClusterImpl.java:304)
        at org.apache.hadoop.ozone.MiniOzoneClusterImpl.shutdown(MiniOzoneClusterImpl.java:446)
        at org.apache.hadoop.ozone.om.TestSnapshotBackgroundServices.shutdown(TestSnapshotBackgroundServices.java:199)

Issue Links

• is related to: HDDS-9455 Intermittent failure in TestSnapshotBackgroundServices
• is related to: HDDS-10185 Revisit LogAppenderWaitTimeMin setting in OzoneManagerRatisServerConfig
• links to: GitHub Pull Request #6026