[HDDS-4852] acceptance (secure) fails frequently at `ozone freon ockg -n1 -t1 -p container` in container.robot

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Component/s: test
• Assignee: Unassigned
• Reporter: Siyao Meng
• Created: 20/Feb/21 05:41
• Updated: 20/Feb/21 12:33
• Resolved: 20/Feb/21 12:33

Description

It's quite annoying this fails often in CIs for a while. Very flaky.

https://github.com/apache/ozone/blob/bc3e3e5f62a871c50f6261534424c67a53713d22/hadoop-ozone/dist/src/main/smoketest/admincli/container.robot#L29

https://github.com/apache/ozone/runs/1939233926 , log.html

2021-02-20 00:24:19,978 [Thread-5] INFO freon.ProgressBar: Progress: 0.00 % (0 out of 1)
2021-02-20 00:24:20,294 [main] INFO rpc.RpcClient: Creating Volume: vol1, with testuser/[e-mail] as owner and space quota set to -1 bytes, counts quota set to -1
2021-02-20 00:24:20,338 [main] INFO rpc.RpcClient: Creating Bucket: vol1/bucket1, with Versioning false and Storage Type set to DISK and Encryption set to false 
2021-02-20 00:24:20,416 [pool-1-thread-1] ERROR freon.BaseFreonGenerator: Error on executing task 0
INTERNAL_ERROR org.apache.hadoop.ozone.om.exceptions.OMException: Allocated 0 blocks. Requested 1 blocks
	at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:604)
	at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.openKey(OzoneManagerProtocolClientSideTranslatorPB.java:595)
	at org.apache.hadoop.ozone.client.rpc.RpcClient.createKey(RpcClient.java:716)
	at org.apache.hadoop.ozone.client.OzoneBucket.createKey(OzoneBucket.java:491)
	at org.apache.hadoop.ozone.freon.OzoneClientKeyGenerator.lambda$createKey$0(OzoneClientKeyGenerator.java:114)
	at com.codahale.metrics.Timer.time(Timer.java:101)
	at org.apache.hadoop.ozone.freon.OzoneClientKeyGenerator.createKey(OzoneClientKeyGenerator.java:113)
	at org.apache.hadoop.ozone.freon.BaseFreonGenerator.tryNextTask(BaseFreonGenerator.java:188)
	at org.apache.hadoop.ozone.freon.BaseFreonGenerator.taskLoop(BaseFreonGenerator.java:168)
	at org.apache.hadoop.ozone.freon.BaseFreonGenerator.lambda$startTaskRunners$0(BaseFreonGenerator.java:151)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
2021-02-20 00:24:20,979 [Thread-5] INFO freon.ProgressBar: Progress: 0.00 % (0 out of 1)
2021-02-20 00:24:21,979 [Thread-5] INFO freon.ProgressBar: Progress: 100.00 % (1 out of 1)
One ore more freon test is failed.
2021-02-20 00:24:22,002 [Thread-4] INFO metrics: type=TIMER, name=key-create, count=1, min=26.521721, max=26.521721, mean=26.521721, stddev=0.0, median=26.521721, p75=26.521721, p95=26.521721, p98=26.521721, p99=26.521721, p999=26.521721, mean_rate=0.6144826916810641, m1=0.0, m5=0.0, m15=0.0, rate_unit=events/second, duration_unit=milliseconds
2021-02-20 00:24:22,006 [Thread-4] INFO freon.BaseFreonGenerator: Total execution time (sec): 3
2021-02-20 00:24:22,006 [Thread-4] INFO freon.BaseFreonGenerator: Failures: 1
2021-02-20 00:24:22,006 [Thread-4] INFO freon.BaseFreonGenerator: Successful executions: 0	

The OMException: Allocated 0 blocks. Requested 1 blocks seems to be the culprit.

Issue Links

• duplicates: HDDS-4850 Intermittent failure in ozonesecure due to unable to allocate block