Uploaded image for project: 'Hadoop Distributed Data Store'
  1. Hadoop Distributed Data Store
  2. HDDS-934

Freon run hung on non-functional pipeline

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.4.0
    • Component/s: Ozone Client
    • Labels:
      None
    • Target Version/s:

      Description

      steps taken:

      1. created docker cluster with 3 datanodes where datanodes are not able to communicate with each other.
      2. ran freon to write 1 key

      freon run threw following exception but it did not terminate.

      exception:

      ------------------

       

      2018-12-18 10:38:06 INFO RandomKeyGenerator:227 - Number of Threads: 10
      2018-12-18 10:38:06 INFO RandomKeyGenerator:233 - Number of Volumes: 1.
      2018-12-18 10:38:06 INFO RandomKeyGenerator:234 - Number of Buckets per Volume: 1.
      2018-12-18 10:38:06 INFO RandomKeyGenerator:235 - Number of Keys per Bucket: 1.
      2018-12-18 10:38:06 INFO RandomKeyGenerator:236 - Key size: 10240 bytes
      2018-12-18 10:38:06 INFO RandomKeyGenerator:266 - Starting progress bar Thread.
      0.00% |█ | 0/1 Time: 0:00:002018-12-18 10:38:06 INFO RpcClient:250 - Creating Volume: vol-0-74492, with hadoop as owner and quota set to 1152921504606846976 bytes.
      2018-12-18 10:38:06 INFO RpcClient:379 - Creating Bucket: vol-0-74492/bucket-0-16002, with Versioning false and Storage Type set to DISK
       0.00% |█ | 0/1 Time: 0:02:402018-12-18 10:40:46 ERROR ChunkGroupOutputStream:275 - Try to allocate more blocks for write failed, already allocated 0 blocks for this write.
      2018-12-18 10:40:46 ERROR RandomKeyGenerator:624 - Exception while adding key: key-0-28925 in bucket: org.apache.hadoop.ozone.client.OzoneBucket@1675c402 of volume: org.apache.hadoop.ozone.client.OzoneVolume@5b6bfafd.
      java.io.IOException: Allocate block failed, error:INTERNAL_ERROR
       at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.allocateBlock(OzoneManagerProtocolClientSideTranslatorPB.java:620)
       at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.allocateNewBlock(ChunkGroupOutputStream.java:437)
       at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.handleWrite(ChunkGroupOutputStream.java:272)
       at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.handleException(ChunkGroupOutputStream.java:377)
       at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.handleFlushOrClose(ChunkGroupOutputStream.java:473)
       at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.handleFlushOrClose(ChunkGroupOutputStream.java:474)
       at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.handleFlushOrClose(ChunkGroupOutputStream.java:474)
       at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.handleFlushOrClose(ChunkGroupOutputStream.java:474)
       at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.handleWrite(ChunkGroupOutputStream.java:309)
       at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.write(ChunkGroupOutputStream.java:255)
       at org.apache.hadoop.ozone.client.io.OzoneOutputStream.write(OzoneOutputStream.java:49)
       at java.io.OutputStream.write(OutputStream.java:75)
       at org.apache.hadoop.ozone.freon.RandomKeyGenerator$OfflineProcessor.run(RandomKeyGenerator.java:606)
       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
       at java.util.concurrent.FutureTask.run(FutureTask.java:266)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       at java.lang.Thread.run(Thread.java:748)
       0.00% |█ | 0/1 Time: 0:11:01
       0.00% |█ | 0/1 Time: 0:11:19
      

      Here is the jstack for the freon process :

      jstack.txt

       

        Attachments

        1. HDDS-934.000.patch
          1 kB
          Nanda kumar
        2. jstack.txt
          16 kB
          Nilotpal Nandi

          Activity

            People

            • Assignee:
              nanda Nanda kumar
              Reporter:
              nilotpalnandi Nilotpal Nandi
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: