Uploaded image for project: 'Hadoop Distributed Data Store'
  1. Hadoop Distributed Data Store
  2. HDDS-1613

Opening of rocksDB in datanode fails with "No locks available"

    XMLWordPrintableJSON

    Details

    • Target Version/s:
    • Sprint:
      HDDS Biscayne

      Description

      Block read fails with

      org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException: Unable to find the block with bcsID 11777 .Container 68 bcsId is 0.
              at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.validateContainerResponse(ContainerProtocolCalls.java:573)
              at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.getBlock(ContainerProtocolCalls.java:120)
              at org.apache.hadoop.ozone.client.io.KeyInputStream$ChunkInputStreamEntry.initializeBlockInputStream(KeyInputStream.java:295)
              at org.apache.hadoop.ozone.client.io.KeyInputStream$ChunkInputStreamEntry.getStream(KeyInputStream.java:265)
              at org.apache.hadoop.ozone.client.io.KeyInputStream$ChunkInputStreamEntry.access$000(KeyInputStream.java:229)
              at org.apache.hadoop.ozone.client.io.KeyInputStream.getStreamEntry(KeyInputStream.java:107)
              at org.apache.hadoop.ozone.client.io.KeyInputStream.read(KeyInputStream.java:140)
              at org.apache.hadoop.ozone.client.io.OzoneInputStream.read(OzoneInputStream.java:47)
              at java.io.InputStream.read(InputStream.java:101)
              at org.apache.hadoop.ozone.MiniOzoneLoadGenerator.load(MiniOzoneLoadGenerator.java:114)
              at org.apache.hadoop.ozone.MiniOzoneLoadGenerator.lambda$startIO$0(MiniOzoneLoadGenerator.java:147)
              at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      

      Looking at the 3 datanodes, the containers are in bcs id of 11748, 11748 and 0.

      2019-05-30 08:28:05,348 INFO  keyvalue.KeyValueHandler (ContainerUtils.java:logAndReturnError(146)) - Operation: GetBlock : Trace ID: 93a2a596076d2ee4:93a2a596076d2ee4:0:0 : Message: Unable to find the block with bcsID 11777 .Container 68 bcsId is 11748. : Result: UNKNOWN_BCSID
      
      
      2019-05-30 08:28:05,363 INFO  keyvalue.KeyValueHandler (ContainerUtils.java:logAndReturnError(146)) - Operation: GetBlock : Trace ID: 93a2a596076d2ee4:93a2a596076d2ee4:0:0 : Message: Unable to find the block with bcsID 11777 .Container 68 bcsId is 11748. : Result: UNKNOWN_BCSID
      
      
      2019-05-30 08:28:05,377 INFO  keyvalue.KeyValueHandler (ContainerUtils.java:logAndReturnError(146)) - Operation: GetBlock : Trace ID: 93a2a596076d2ee4:93a2a596076d2ee4:0:0 : Message: Unable to find the block with bcsID 11777 .Container 68 bcsId is 0. : Result: UNKNOWN_BCSID
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                msingh Mukul Kumar Singh
                Reporter:
                msingh Mukul Kumar Singh
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h 20m
                  1h 20m