[RATIS-538] OverlappingFileLockException exception in datanodes after restart in MiniOzoneChaosCluster

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 0.3.0
• Labels: MiniOzoneChaosCluster ozone
• Assignee: Unassigned
• Reporter: Mukul Kumar Singh
• Created: 17/Apr/19 04:47
• Updated: 29/Jun/22 17:57
• Resolved: 29/Jun/22 17:57

Description

Following exception can be seen in MiniOzoneChaosCluster after Datanode restarts.

java.io.IOException: Failed to lock storage /Users/msingh/code/apache/ozone/oz_new1/hadoop-ozone/integration-test/target/test/data/MiniOzoneClusterImpl-6e7bf33e-55b8-4437-8a0d-0da730510c70/datanode-4/data/ratis/49740cb4-b1cf-418a-af8e-1b28e0bf6cbc. The directory is already locked
        at org.apache.ratis.server.storage.RaftStorageDirectory.tryLock(RaftStorageDirectory.java:334)
        at org.apache.ratis.server.storage.RaftStorageDirectory.lock(RaftStorageDirectory.java:291)
        at org.apache.ratis.server.storage.RaftStorageDirectory.analyzeStorage(RaftStorageDirectory.java:264)
        at org.apache.ratis.server.storage.RaftStorage.analyzeAndRecoverStorage(RaftStorage.java:91)
        at org.apache.ratis.server.storage.RaftStorage.<init>(RaftStorage.java:59)
        at org.apache.ratis.server.impl.ServerState.<init>(ServerState.java:106)
        at org.apache.ratis.server.impl.RaftServerImpl.<init>(RaftServerImpl.java:101)
        at org.apache.ratis.server.impl.RaftServerProxy.lambda$newRaftServerImpl$2(RaftServerProxy.java:208)
        at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.channels.OverlappingFileLockException
        at sun.nio.ch.SharedFileLockTable.checkList(FileLockTable.java:255)
        at sun.nio.ch.SharedFileLockTable.add(FileLockTable.java:152)
        at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:1108)
        at java.nio.channels.FileChannel.tryLock(FileChannel.java:1155)
        at org.apache.ratis.server.storage.RaftStorageDirectory.tryLock(RaftStorageDirectory.java:322)