[HDDS-1613] Opening of rocksDB in datanode fails with "No locks available"

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.4.0
• Fix Version/s: 0.5.0
• Component/s: Ozone Client
• Labels: MiniOzoneChaosCluster pull-request-available
• Target Version/s: 0.5.0
• Sprint: Done
• Assignee: Mukul Kumar Singh
• Reporter: Mukul Kumar Singh
• Created: 30/May/19 09:43
• Updated: 24/Feb/20 20:55
• Resolved: 03/Jun/19 20:44

Description

Block read fails with

org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException: Unable to find the block with bcsID 11777 .Container 68 bcsId is 0.
        at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.validateContainerResponse(ContainerProtocolCalls.java:573)
        at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.getBlock(ContainerProtocolCalls.java:120)
        at org.apache.hadoop.ozone.client.io.KeyInputStream$ChunkInputStreamEntry.initializeBlockInputStream(KeyInputStream.java:295)
        at org.apache.hadoop.ozone.client.io.KeyInputStream$ChunkInputStreamEntry.getStream(KeyInputStream.java:265)
        at org.apache.hadoop.ozone.client.io.KeyInputStream$ChunkInputStreamEntry.access$000(KeyInputStream.java:229)
        at org.apache.hadoop.ozone.client.io.KeyInputStream.getStreamEntry(KeyInputStream.java:107)
        at org.apache.hadoop.ozone.client.io.KeyInputStream.read(KeyInputStream.java:140)
        at org.apache.hadoop.ozone.client.io.OzoneInputStream.read(OzoneInputStream.java:47)
        at java.io.InputStream.read(InputStream.java:101)
        at org.apache.hadoop.ozone.MiniOzoneLoadGenerator.load(MiniOzoneLoadGenerator.java:114)
        at org.apache.hadoop.ozone.MiniOzoneLoadGenerator.lambda$startIO$0(MiniOzoneLoadGenerator.java:147)
        at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Looking at the 3 datanodes, the containers are in bcs id of 11748, 11748 and 0.

2019-05-30 08:28:05,348 INFO  keyvalue.KeyValueHandler (ContainerUtils.java:logAndReturnError(146)) - Operation: GetBlock : Trace ID: 93a2a596076d2ee4:93a2a596076d2ee4:0:0 : Message: Unable to find the block with bcsID 11777 .Container 68 bcsId is 11748. : Result: UNKNOWN_BCSID


2019-05-30 08:28:05,363 INFO  keyvalue.KeyValueHandler (ContainerUtils.java:logAndReturnError(146)) - Operation: GetBlock : Trace ID: 93a2a596076d2ee4:93a2a596076d2ee4:0:0 : Message: Unable to find the block with bcsID 11777 .Container 68 bcsId is 11748. : Result: UNKNOWN_BCSID


2019-05-30 08:28:05,377 INFO  keyvalue.KeyValueHandler (ContainerUtils.java:logAndReturnError(146)) - Operation: GetBlock : Trace ID: 93a2a596076d2ee4:93a2a596076d2ee4:0:0 : Message: Unable to find the block with bcsID 11777 .Container 68 bcsId is 0. : Result: UNKNOWN_BCSID

Issue Links

• duplicates: HDDS-1614 Container Missing in the datanode after restart
• links to: GitHub Pull Request #876