Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.9
    • Fix Version/s: 0.9
    • Component/s: None
    • Labels:
      None
    • Environment:

      gentoo linux on c1.medium aws instance

      Description

      I am using couchrest to transform a document from one database to another (millions of records). The memory usage continues to increase till all the memory is used. The process occasionally fails, but usually completes. I dont know enough about ruby/couchrest/couchdb to be able to tell you where the memory is increasing from - but if you give me some direction, I will be happy to take a look.

      If I comment out the save, it seems ok.

      Here is the code:

      --------------------------------

      require 'rubygems'
      require 'couchrest'

      @db = CouchRest.database!("http://127.0.0.1:5984/xfpds_2008")
      @db2 = CouchRest.database!("http://127.0.0.1:5984/fpds_2008")

      @streamer = CouchRest::Streamer.new(@db)
      @streamer.view("_all_docs") do |row|
      begin

      doc = @db.get(row["id"])

      1. remove id so the new database gets fresh document
        doc.delete("_id")
        doc.delete("_rev")
      1. change badgerfish notation to assign $ key value to parent
        doc["amounts"]["obligatedAmount"] = doc["amounts"]["obligatedAmount"]["$"]
        doc["amounts"]["baseAndAllOptionsValue"] = doc["amounts"]["baseAndAllOptionsValue"]["$"]
        doc["amounts"]["baseAndExercisedOptionsValue"] = doc["amounts"]["baseAndExercisedOptionsValue"]["$"]

      ...... A whole bunch of fields

      1. save the document using bulk save
        response = @db2.save_doc(doc,true)

      rescue

      1. if streamer ends, save last few documents
        @db2.bulk_save
        end

      end

        Activity

        Rohit Amarnath created issue -
        Hide
        Jan Lehnardt added a comment -

        Can you add some information about what exactly is eating the memory? Ruby or CouchDB?

        Show
        Jan Lehnardt added a comment - Can you add some information about what exactly is eating the memory? Ruby or CouchDB?
        Hide
        Rohit Amarnath added a comment -

        Not sure, could you point me in the direction of some tool I could install and run to give you that info?

        Show
        Rohit Amarnath added a comment - Not sure, could you point me in the direction of some tool I could install and run to give you that info?
        Hide
        Rohit Amarnath added a comment -

        I added the following code to the ruby program:

        memory_usage = `ps -o rss= -p #

        {Process.pid}`.to_i # physical memory in kilobytes
        memory_usage = memory_usage + `ps -o vsz= -p #{Process.pid}

        `.to_i # Virtual memory in kilobytes

        if memory_usage > memory_last
        puts "memory increased by #

        {memory_usage - memory_last} KB to #{memory_usage/1024} MB"
        elsif memory_usage < memory_last
        puts "memory decreased by #{memory_usage - memory_last}

        KB to #

        {memory_usage/1024}

        MB"
        end

        memory_last = memory_usage

        The memory for ruby goes up to about 260MB and hovers between 230MB and 260MB, but the memory usage on the system continues to increase to the max. Everytime it does a bulk load the memory goes up by 30MB, but then the garbage collector kicks in and reduces it back to 230MB.

        Show
        Rohit Amarnath added a comment - I added the following code to the ruby program: memory_usage = `ps -o rss= -p # {Process.pid}`.to_i # physical memory in kilobytes memory_usage = memory_usage + `ps -o vsz= -p #{Process.pid} `.to_i # Virtual memory in kilobytes if memory_usage > memory_last puts "memory increased by # {memory_usage - memory_last} KB to #{memory_usage/1024} MB" elsif memory_usage < memory_last puts "memory decreased by #{memory_usage - memory_last} KB to # {memory_usage/1024} MB" end memory_last = memory_usage The memory for ruby goes up to about 260MB and hovers between 230MB and 260MB, but the memory usage on the system continues to increase to the max. Everytime it does a bulk load the memory goes up by 30MB, but then the garbage collector kicks in and reduces it back to 230MB.
        Hide
        Rohit Amarnath added a comment -

        Looks like using htop, I was able to identify the memory usage as caches. So am closing this.

        Show
        Rohit Amarnath added a comment - Looks like using htop, I was able to identify the memory usage as caches. So am closing this.
        Rohit Amarnath made changes -
        Field Original Value New Value
        Status Open [ 1 ] Closed [ 6 ]
        Resolution Fixed [ 1 ]

          People

          • Assignee:
            Unassigned
            Reporter:
            Rohit Amarnath
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development