Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
Description
steps taken:
- created docker cluster with 3 datanodes where datanodes are not able to communicate with each other.
- ran freon to write 1 key
freon run threw following exception but it did not terminate.
exception:
------------------
2018-12-18 10:38:06 INFO RandomKeyGenerator:227 - Number of Threads: 10 2018-12-18 10:38:06 INFO RandomKeyGenerator:233 - Number of Volumes: 1. 2018-12-18 10:38:06 INFO RandomKeyGenerator:234 - Number of Buckets per Volume: 1. 2018-12-18 10:38:06 INFO RandomKeyGenerator:235 - Number of Keys per Bucket: 1. 2018-12-18 10:38:06 INFO RandomKeyGenerator:236 - Key size: 10240 bytes 2018-12-18 10:38:06 INFO RandomKeyGenerator:266 - Starting progress bar Thread. 0.00% |█ | 0/1 Time: 0:00:002018-12-18 10:38:06 INFO RpcClient:250 - Creating Volume: vol-0-74492, with hadoop as owner and quota set to 1152921504606846976 bytes. 2018-12-18 10:38:06 INFO RpcClient:379 - Creating Bucket: vol-0-74492/bucket-0-16002, with Versioning false and Storage Type set to DISK 0.00% |█ | 0/1 Time: 0:02:402018-12-18 10:40:46 ERROR ChunkGroupOutputStream:275 - Try to allocate more blocks for write failed, already allocated 0 blocks for this write. 2018-12-18 10:40:46 ERROR RandomKeyGenerator:624 - Exception while adding key: key-0-28925 in bucket: org.apache.hadoop.ozone.client.OzoneBucket@1675c402 of volume: org.apache.hadoop.ozone.client.OzoneVolume@5b6bfafd. java.io.IOException: Allocate block failed, error:INTERNAL_ERROR at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.allocateBlock(OzoneManagerProtocolClientSideTranslatorPB.java:620) at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.allocateNewBlock(ChunkGroupOutputStream.java:437) at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.handleWrite(ChunkGroupOutputStream.java:272) at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.handleException(ChunkGroupOutputStream.java:377) at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.handleFlushOrClose(ChunkGroupOutputStream.java:473) at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.handleFlushOrClose(ChunkGroupOutputStream.java:474) at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.handleFlushOrClose(ChunkGroupOutputStream.java:474) at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.handleFlushOrClose(ChunkGroupOutputStream.java:474) at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.handleWrite(ChunkGroupOutputStream.java:309) at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.write(ChunkGroupOutputStream.java:255) at org.apache.hadoop.ozone.client.io.OzoneOutputStream.write(OzoneOutputStream.java:49) at java.io.OutputStream.write(OutputStream.java:75) at org.apache.hadoop.ozone.freon.RandomKeyGenerator$OfflineProcessor.run(RandomKeyGenerator.java:606) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 0.00% |█ | 0/1 Time: 0:11:01 0.00% |█ | 0/1 Time: 0:11:19
Here is the jstack for the freon process :