[HDDS-2330] Random key generator can get stuck

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.5.0
• Component/s: freon
• Labels: pull-request-available
• Target Version/s: 0.5.0
• Assignee: Attila Doroszlai
• Reporter: Attila Doroszlai
• Created: 18/Oct/19 10:53
• Updated: 24/Feb/20 20:54
• Resolved: 22/Oct/19 02:41

Description

Freon's random key generator can get stuck waiting for completion (without any hint to what's happening) if object creation encounters any non-IOException.

Steps to reproduce:

1. Start Ozone cluster with 1 datanode
2. Start Freon (5K keys of size 1MB)

Result: after a few hundred keys progress stops.

$ docker-compose exec scm ozone freon rk --numOfThreads 1 --numOfVolumes 1 --numOfBuckets 1 --replicationType RATIS --factor ONE --keySize $(echo '2^20' | bc -lq) --numOfKeys $(echo '5 * 2^10' | bc -lq) --bufferSize $(echo '2^16' | bc -lq)
2019-10-18 10:44:45,224 INFO impl.MetricsConfig: Loaded properties from hadoop-metrics2.properties
2019-10-18 10:44:45,381 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
2019-10-18 10:44:45,381 INFO impl.MetricsSystemImpl: ozone-freon metrics system started
2019-10-18 10:44:47,140 [main] INFO       - Number of Threads: 1
2019-10-18 10:44:47,145 [main] INFO       - Number of Volumes: 1.
2019-10-18 10:44:47,146 [main] INFO       - Number of Buckets per Volume: 1.
2019-10-18 10:44:47,146 [main] INFO       - Number of Keys per Bucket: 5120.
2019-10-18 10:44:47,147 [main] INFO       - Key size: 1048576 bytes
2019-10-18 10:44:47,147 [main] INFO       - Buffer size: 65536 bytes
2019-10-18 10:44:47,147 [main] INFO       - validateWrites : false
2019-10-18 10:44:47,151 [main] INFO       - Starting progress bar Thread.
...
 7.07% |????????                                                                                             |  362/5120 

Issue Links

• links to: GitHub Pull Request #53