VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Blocker
    • Resolution: Fixed
    • 0.19.0, 0.20.0
    • 0.18.3, 0.19.1
    • None
    • None
    • Reviewed
    • Hide
      fix memory leak of user/group information in fuse-dfs

      Show
      fix memory leak of user/group information in fuse-dfs

    Description

      I am running a process that needs to crawl a tree structure containing ~10K images, copy the images to the local disk, process these images, and copy them back to HDFS.

      My problem is the following : after about 10h of processing, the processes crash, complaining about a std::bad_alloc exception (I use hadoop pipes to run existing software). When running fuse_dfs in debug mode, I get an outOfMemoryError, telling that there is no more room in the heap.

      While the process is running, using top or ps, I notice that fuse is using up an increasing amount of memory, until some limit is reached. At that point , the memory used is oscillating. I suppose that this is due to the use of the virtual memory.

      This leads me to the conclusion that there is some memory leak in fuse_dfs, since the only other programs running are Hadoop and the existing software, both thoroughly tested in the past.

      My problem is that my knowledge concerning memory leak tracking is rather limited, so I will need some instructions to get more insight concerning this issue.

      Thank you

      Attachments

        1. patch-hadoop4635.test_18
          32 kB
          Pete Wyckoff
        2. patch-hadoop4635.test_19
          61 kB
          Pete Wyckoff
        3. TEST-TestFuseDFS.txt-4635
          36 kB
          Pete Wyckoff
        4. HADOOP-4635.txt.0.18
          3 kB
          Pete Wyckoff
        5. HADOOP-4635.txt.0.19
          2 kB
          Pete Wyckoff

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            wyckoff Pete Wyckoff
            mofleury Marc-Olivier Fleury
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Issue deployment