HDDS-1532 modified the concurrent framework usage of Freon (RandomKeyGenerator).
The new approach uses separated tasks (Runnable) to create the volumes/buckets/keys.
Unfortunately it doesn't work very well in some cases.
- When Freon starts it creates an executor with fixed number of threads (10)
- The first loop submits numOfVolumes (10) VolumeProcessor tasks to the executor
- The 10 threads starts to execute the 10 VolumeProcessor tasks
- Each VolumeProcessor tasks creates numOfBuckets (1000) BucketProcessor tasks. All together 10000 tasks are submitted to the executor.
- The 10 threads starts to execute the first 10 BucketProcessor tasks, they starts to create the KeyProcessor tasks: 500 000 * 10 tasks are submitted.
- At this point of the time no keys are generated, but the next 10 BucketProcessor tasks are started to execute..
- To execute the first key creation we should process all the BucketProcessor tasks which means that all the Key creation tasks (10 * 1000 * 500 000) are created and added to the executor
- Which requires a huge amount of time and memory