Uploaded image for project: 'Hadoop Distributed Data Store'
  1. Hadoop Distributed Data Store
  2. HDDS-1785

OOM error in Freon due to the concurrency handling

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.4.2
    • Component/s: None
    • Target Version/s:
    • Sprint:
      HDDS Biscayne

      Description

      HDDS-1532 modified the concurrent framework usage of Freon (RandomKeyGenerator).

      The new approach uses separated tasks (Runnable) to create the volumes/buckets/keys.

      Unfortunately it doesn't work very well in some cases.

      1. When Freon starts it creates an executor with fixed number of threads (10)
      2. The first loop submits numOfVolumes (10) VolumeProcessor tasks to the executor
      3. The 10 threads starts to execute the 10 VolumeProcessor tasks
      4. Each VolumeProcessor tasks creates numOfBuckets (1000) BucketProcessor tasks. All together 10000 tasks are submitted to the executor.
      5. The 10 threads starts to execute the first 10 BucketProcessor tasks, they starts to create the KeyProcessor tasks: 500 000 * 10 tasks are submitted.
      6. At this point of the time no keys are generated, but the next 10 BucketProcessor tasks are started to execute..
      7. To execute the first key creation we should process all the BucketProcessor tasks which means that all the Key creation tasks (10 * 1000 * 500 000) are created and added to the executor
      8. Which requires a huge amount of time and memory

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                adoroszlai Attila Doroszlai
                Reporter:
                elek Marton Elek
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 50m
                  50m