Uploaded image for project: 'Apache Ozone'
  1. Apache Ozone
  2. HDDS-1785

OOM error in Freon due to the concurrency handling

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • None
    • 0.5.0
    • None
    • Done

    Description

      HDDS-1532 modified the concurrent framework usage of Freon (RandomKeyGenerator).

      The new approach uses separated tasks (Runnable) to create the volumes/buckets/keys.

      Unfortunately it doesn't work very well in some cases.

      1. When Freon starts it creates an executor with fixed number of threads (10)
      2. The first loop submits numOfVolumes (10) VolumeProcessor tasks to the executor
      3. The 10 threads starts to execute the 10 VolumeProcessor tasks
      4. Each VolumeProcessor tasks creates numOfBuckets (1000) BucketProcessor tasks. All together 10000 tasks are submitted to the executor.
      5. The 10 threads starts to execute the first 10 BucketProcessor tasks, they starts to create the KeyProcessor tasks: 500 000 * 10 tasks are submitted.
      6. At this point of the time no keys are generated, but the next 10 BucketProcessor tasks are started to execute..
      7. To execute the first key creation we should process all the BucketProcessor tasks which means that all the Key creation tasks (10 * 1000 * 500 000) are created and added to the executor
      8. Which requires a huge amount of time and memory

      Attachments

        Issue Links

          Activity

            People

              adoroszlai Attila Doroszlai
              elek Marton Elek
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 50m
                  50m