[HDDS-8925] BaseFreonGenerator may not complete with only failed attempt

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.4.0
• Component/s: freon
• Labels: pull-request-available
• Target Version/s: 1.4.0
• Assignee: Attila Doroszlai
• Reporter: Attila Doroszlai
• Created: 25/Jun/23 11:40
• Updated: 27/Jun/23 05:07
• Resolved: 27/Jun/23 05:07

Description

• Worker thread sets completed=true on attempting n+1. task.
• Progress bar is terminated if there are failed attempts, shutdown if not.

The problem happens if there are tasks in progress when progressbar shutdown is initiated AND one of these in-progress tasks is the first to fail.

repro

$ docker-compose exec -T s3g ozone freon s3kg -n 1 -t 10 -b no-such-bucket
...
2023-06-25 13:30:46,493 [main] INFO freon.BaseFreonGenerator: Executing test with prefix p0l2uvfgg7 and number-of-tests 1
2023-06-25 13:30:46,507 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 % (0 out of 1)
2023-06-25 13:30:46,792 [pool-2-thread-1] ERROR freon.BaseFreonGenerator: Error on executing task 0
com.amazonaws.SdkClientException: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))
	at com.amazonaws.auth.EnvironmentVariableCredentialsProvider.getCredentials(EnvironmentVariableCredentialsProvider.java:50)
	...
	at org.apache.hadoop.ozone.freon.S3KeyGenerator.createKey(S3KeyGenerator.java:111)
	at org.apache.hadoop.ozone.freon.BaseFreonGenerator.tryNextTask(BaseFreonGenerator.java:220)
	at org.apache.hadoop.ozone.freon.BaseFreonGenerator.taskLoop(BaseFreonGenerator.java:200)
	at org.apache.hadoop.ozone.freon.BaseFreonGenerator.lambda$startTaskRunners$0(BaseFreonGenerator.java:174)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
2023-06-25 13:30:47,509 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 % (0 out of 1)
2023-06-25 13:30:48,510 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 % (0 out of 1)
2023-06-25 13:30:49,511 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 % (0 out of 1)
2023-06-25 13:30:50,513 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 % (0 out of 1)
2023-06-25 13:30:51,514 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 % (0 out of 1)
2023-06-25 13:30:52,515 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 % (0 out of 1)
2023-06-25 13:30:53,517 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 % (0 out of 1)
2023-06-25 13:30:54,518 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 % (0 out of 1)
2023-06-25 13:30:55,519 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 % (0 out of 1)
2023-06-25 13:30:56,520 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 % (0 out of 1)
...

Issue Links

• links to: GitHub Pull Request #4975