Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Component/s: Server
    • Labels:
      None
    • Environment:
      deltacloud-core-1.0.0-4.fc17.noarch (though same issue with dc-core from git)
      sinatra (1.3.2)
      sinatra-rabbit (1.0.6)
      Fedora 17

      Description

      Memory used by deltacloudd process increases with time. After 12 hours, it's common it uses ~4G of virt memory and >1G of physical memory (only conductor's dbomatic script which periodically checks running instances every minute was running).

      You can reproduce the memory leak by this simple script:
      #!/usr/bin/ruby
      require 'rubygems'
      require 'deltacloud'
      1000.times do |i|

      { DeltaCloud.new('mockuser', 'mockpassword', 'http://localhost:3002/api') }

      Used memory will increase by >10M (both virt and physical). The problem is probably with lambda in check_capability usage in lib/deltacloud/collections/* files:
      check_capability :for => lambda

      { |m| driver.respond_to? m }

      if the lambda is replaced by true or other static value, the memory usage remains constant.
      I see that in git the 'check_capability' call was replaced by 'set' call, this change has no influence on this problem.

      Also it's weird that after starting deltacloudd, used memory is 233m/33m (virt/res), but after first request, used memory jumps to 776m/90m (virt/res) - this is far more than on Fedora 16, where used memory remains around 160m/50m.

        Activity

        Jan Provaznik created issue -
        Hide
        Michal Fojtik added a comment -

        Hi,

        After some debugging we tried this patch that seems to solve the problem of memory growing. However, the memory usage after first request still jump to 1.5gb.

        diff --git a/server/lib/deltacloud/helpers/driver_helper.rb b/server/lib/deltacloud/helpers/driver_helper.r
        index 5c5e7c6..96fd1d7 100644
        — a/server/lib/deltacloud/helpers/driver_helper.rb
        +++ b/server/lib/deltacloud/helpers/driver_helper.rb
        @@ -49,8 +49,12 @@ module Deltacloud::Helpers
        def driver
        $:.unshift File.join(File.dirname(_FILE_), '..', '..')
        begin

        • require_relative driver_source_name
        • driver_class
          + @@driver ||= nil
          + unless @@driver
          + puts "Loading driver..."
          + require_relative driver_source_name
          + end
          + @@driver ||= driver_class
          rescue LoadError => e
          raise "[ERROR] The driver '# {driver_name}

          ' is unknown or not installed (#

          {driver_source_name}

          )\n"
          "\n#

          {e.message}

          \n"

        Show
        Michal Fojtik added a comment - Hi, After some debugging we tried this patch that seems to solve the problem of memory growing. However, the memory usage after first request still jump to 1.5gb. diff --git a/server/lib/deltacloud/helpers/driver_helper.rb b/server/lib/deltacloud/helpers/driver_helper.r index 5c5e7c6..96fd1d7 100644 — a/server/lib/deltacloud/helpers/driver_helper.rb +++ b/server/lib/deltacloud/helpers/driver_helper.rb @@ -49,8 +49,12 @@ module Deltacloud::Helpers def driver $:.unshift File.join(File.dirname(_ FILE _), '..', '..') begin require_relative driver_source_name driver_class + @@driver ||= nil + unless @@driver + puts "Loading driver..." + require_relative driver_source_name + end + @@driver ||= driver_class rescue LoadError => e raise " [ERROR] The driver '# {driver_name} ' is unknown or not installed (# {driver_source_name} )\n" "\n# {e.message} \n"
        Hide
        Michal Fojtik added a comment -

        (remove the 'puts'

        Show
        Michal Fojtik added a comment - (remove the 'puts'
        Hide
        Michal Fojtik added a comment -

        I modified the patch to work nicely with our driver switching. Patch is on the list. Once you confirm it works for you I'll push it and apply
        as hot-fix for F17 RPM.

        Show
        Michal Fojtik added a comment - I modified the patch to work nicely with our driver switching. Patch is on the list. Once you confirm it works for you I'll push it and apply as hot-fix for F17 RPM.
        Michal Fojtik made changes -
        Field Original Value New Value
        Assignee Michal Fojtik [ mfojtik ]
        Hide
        Jan Provaznik added a comment -

        I can confirm that the original patch in first comment fixes memory leak. But I don't see the modified version of the patch or link to it (if you need confirm for it).

        Show
        Jan Provaznik added a comment - I can confirm that the original patch in first comment fixes memory leak. But I don't see the modified version of the patch or link to it (if you need confirm for it).
        Hide
        Jan Provaznik added a comment -

        The updated patch works fine - no memory leakage, also now it's significantly faster .

        Show
        Jan Provaznik added a comment - The updated patch works fine - no memory leakage, also now it's significantly faster .
        Hide
        Ronelle Landy added a comment -

        Closing issue based on Jan's comments above ... changed was pushed to master in commit 5cefcb418e502b70baf016658471ed0d6a2a995d

        Show
        Ronelle Landy added a comment - Closing issue based on Jan's comments above ... changed was pushed to master in commit 5cefcb418e502b70baf016658471ed0d6a2a995d
        Ronelle Landy made changes -
        Status Open [ 1 ] Closed [ 6 ]
        Resolution Fixed [ 1 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Closed Closed
        12d 5h 20m 1 Ronelle Landy 13/Aug/12 15:53

          People

          • Assignee:
            Michal Fojtik
            Reporter:
            Jan Provaznik
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development