Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-14790

[GLib] Memory leak on creating GArrowData

Agile BoardAttach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      We're having problem with a memory leak in a Ruby script that processes many CSV files. I have written some short scripts do demonstrate the problem: https://gist.github.com/stenlarsson/60b1e4e99416738b41ee30e7ba294214

      The first script, arrow_test_csv.rb, creates a 184 MB CSV file for testing.

      The second script, arrow_memory_leak.rb, then loads the CSV file 10 times using Arrow. It uses the get_process_mem gem to print the memory usage both before and after each iteration. It also invokes the garbage collector on each iteration to ensure the problem is not that Ruby holds on to any objects. This is what it prints on my MacBook Pro using Arrow 6.0.0:

      127577 objects, 34.234375 MB
      127577 objects, 347.625 MB
      127577 objects, 438.7890625 MB
      127577 objects, 457.6953125 MB
      127577 objects, 469.8046875 MB
      127577 objects, 480.88671875 MB
      127577 objects, 487.96484375 MB
      127577 objects, 493.8359375 MB
      127577 objects, 497.671875 MB
      127577 objects, 498.55859375 MB
      127577 objects, 501.42578125 MB
      

      The third script, arrow_memory_leak.py is a Python implementation of the same script. This shows that the problem is not in the Ruby bindings:

      2106 objects, 31.75390625 MB
      2106 objects, 382.28515625 MB
      2106 objects, 549.41796875 MB
      2106 objects, 656.78125 MB
      2106 objects, 679.6875 MB
      2106 objects, 691.9921875 MB
      2106 objects, 708.73828125 MB
      2106 objects, 717.296875 MB
      2106 objects, 724.390625 MB
      2106 objects, 729.19921875 MB
      2106 objects, 734.47265625 MB
      

      I have also tested Arrow 5.0.0 and it has the same problem.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            kou Kouhei Sutou Assign to me
            stenlarsson Sten Larsson
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 40m
                40m

                Slack

                  Issue deployment