Details
-
Bug
-
Status: Resolved
-
Blocker
-
Resolution: Fixed
-
None
-
None
-
Done
Description
HDDS-1532 modified the concurrent framework usage of Freon (RandomKeyGenerator).
The new approach uses separated tasks (Runnable) to create the volumes/buckets/keys.
Unfortunately it doesn't work very well in some cases.
- When Freon starts it creates an executor with fixed number of threads (10)
- The first loop submits numOfVolumes (10) VolumeProcessor tasks to the executor
- The 10 threads starts to execute the 10 VolumeProcessor tasks
- Each VolumeProcessor tasks creates numOfBuckets (1000) BucketProcessor tasks. All together 10000 tasks are submitted to the executor.
- The 10 threads starts to execute the first 10 BucketProcessor tasks, they starts to create the KeyProcessor tasks: 500 000 * 10 tasks are submitted.
- At this point of the time no keys are generated, but the next 10 BucketProcessor tasks are started to execute..
- To execute the first key creation we should process all the BucketProcessor tasks which means that all the Key creation tasks (10 * 1000 * 500 000) are created and added to the executor
- Which requires a huge amount of time and memory
Attachments
Issue Links
- links to