[HDDS-934] Freon run hung on non-functional pipeline

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.4.0
• Component/s: Ozone Client
• Target Version/s: 0.4.0
• Assignee: Nandakumar
• Reporter: Nilotpal Nandi
• Created: 18/Dec/18 10:53
• Updated: 20/Dec/18 06:01
• Resolved: 20/Dec/18 06:01

Description

steps taken:

1. created docker cluster with 3 datanodes where datanodes are not able to communicate with each other.
2. ran freon to write 1 key

freon run threw following exception but it did not terminate.

exception:

------------------

 

2018-12-18 10:38:06 INFO RandomKeyGenerator:227 - Number of Threads: 10
2018-12-18 10:38:06 INFO RandomKeyGenerator:233 - Number of Volumes: 1.
2018-12-18 10:38:06 INFO RandomKeyGenerator:234 - Number of Buckets per Volume: 1.
2018-12-18 10:38:06 INFO RandomKeyGenerator:235 - Number of Keys per Bucket: 1.
2018-12-18 10:38:06 INFO RandomKeyGenerator:236 - Key size: 10240 bytes
2018-12-18 10:38:06 INFO RandomKeyGenerator:266 - Starting progress bar Thread.
0.00% |█ | 0/1 Time: 0:00:002018-12-18 10:38:06 INFO RpcClient:250 - Creating Volume: vol-0-74492, with hadoop as owner and quota set to 1152921504606846976 bytes.
2018-12-18 10:38:06 INFO RpcClient:379 - Creating Bucket: vol-0-74492/bucket-0-16002, with Versioning false and Storage Type set to DISK
 0.00% |█ | 0/1 Time: 0:02:402018-12-18 10:40:46 ERROR ChunkGroupOutputStream:275 - Try to allocate more blocks for write failed, already allocated 0 blocks for this write.
2018-12-18 10:40:46 ERROR RandomKeyGenerator:624 - Exception while adding key: key-0-28925 in bucket: org.apache.hadoop.ozone.client.OzoneBucket@1675c402 of volume: org.apache.hadoop.ozone.client.OzoneVolume@5b6bfafd.
java.io.IOException: Allocate block failed, error:INTERNAL_ERROR
 at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.allocateBlock(OzoneManagerProtocolClientSideTranslatorPB.java:620)
 at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.allocateNewBlock(ChunkGroupOutputStream.java:437)
 at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.handleWrite(ChunkGroupOutputStream.java:272)
 at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.handleException(ChunkGroupOutputStream.java:377)
 at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.handleFlushOrClose(ChunkGroupOutputStream.java:473)
 at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.handleFlushOrClose(ChunkGroupOutputStream.java:474)
 at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.handleFlushOrClose(ChunkGroupOutputStream.java:474)
 at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.handleFlushOrClose(ChunkGroupOutputStream.java:474)
 at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.handleWrite(ChunkGroupOutputStream.java:309)
 at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.write(ChunkGroupOutputStream.java:255)
 at org.apache.hadoop.ozone.client.io.OzoneOutputStream.write(OzoneOutputStream.java:49)
 at java.io.OutputStream.write(OutputStream.java:75)
 at org.apache.hadoop.ozone.freon.RandomKeyGenerator$OfflineProcessor.run(RandomKeyGenerator.java:606)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at java.lang.Thread.run(Thread.java:748)
 0.00% |█ | 0/1 Time: 0:11:01
 0.00% |█ | 0/1 Time: 0:11:19

Here is the jstack for the freon process :

jstack.txt