Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-1953

It may be possible for temporary files to accumulator until the Solr process is shut down

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.4.1, 1.5
    • Fix Version/s: 6.4, 7.0
    • Component/s: update

      Description

      While researching SOLR-1951, the behavior of commons-fileupload in handling multipart form posts came into question. commons-fileupload creates a temporary file for the main content area of such posts, and purportedly has a background thread which cleans up these files. However, Mark Miller discovered that the javadoc in this matter may be incorrect, and that commons-fileupload may in fact just be adding files to the JVM's list of files needing cleanup on exit.

      If so, this will show up in two ways: first, temporary files will accumulate in the java.io.tmpdir area. Second, non-heap memory for the JVM will slowly increase over time (since the file pointers the JVM tracks in this way are not kept in the java heap).

      I will attach a potential fix; however, this ticket should be viewed as a workitem for the need for further research in this area.

      1. SOLR-1953.patch
        11 kB
        Mark Miller
      2. SOLR-1953.patch
        2 kB
        Mark Miller
      3. SOLR-1953.patch
        0.7 kB
        Karl Wright

        Issue Links

          Activity

          Hide
          kwright@metacarta.com Karl Wright added a comment -

          Patch that Mark Miller proposed, if it turns out that his analysis is correct.

          Show
          kwright@metacarta.com Karl Wright added a comment - Patch that Mark Miller proposed, if it turns out that his analysis is correct.
          Hide
          markrmiller@gmail.com Mark Miller added a comment -

          I've got to look at the code - but I wonder if this should be a shared instance that is reused - the cleaner uses a thread that has a special call related to shutting it down. I don't remember all of the specifics, and I have to look at the commons code, but it may not make sense to use a new cleaner on every object creation. Will post back.

          Show
          markrmiller@gmail.com Mark Miller added a comment - I've got to look at the code - but I wonder if this should be a shared instance that is reused - the cleaner uses a thread that has a special call related to shutting it down. I don't remember all of the specifics, and I have to look at the commons code, but it may not make sense to use a new cleaner on every object creation. Will post back.
          Hide
          markrmiller@gmail.com Mark Miller added a comment -

          Yup - def looks to me like we only want one Tracker instance rather than a new one everytime - and we also want to call exitWhenFinished in some clean up code - perhaps in SolrDispatchFilter#destroy()

          Show
          markrmiller@gmail.com Mark Miller added a comment - Yup - def looks to me like we only want one Tracker instance rather than a new one everytime - and we also want to call exitWhenFinished in some clean up code - perhaps in SolrDispatchFilter#destroy()
          Hide
          markrmiller@gmail.com Mark Miller added a comment -

          I think we want something more like this - though it deserves some more thought.

          Show
          markrmiller@gmail.com Mark Miller added a comment - I think we want something more like this - though it deserves some more thought.
          Hide
          kwright@metacarta.com Karl Wright added a comment -

          Looks good to me...

          Show
          kwright@metacarta.com Karl Wright added a comment - Looks good to me...
          Hide
          arafalov Alexandre Rafalovitch added a comment -

          Looking at the code, we may still have this type of memory leaks discussed in 1.4. At least I don't see any registrations with setFileCleaningTracker() as suggested in the patch.

          Show
          arafalov Alexandre Rafalovitch added a comment - Looking at the code, we may still have this type of memory leaks discussed in 1.4. At least I don't see any registrations with setFileCleaningTracker() as suggested in the patch.
          Hide
          moscovig Gilad Moscovitch added a comment - - edited

          Hey. Do you think it is somehow related to the next issue (using solr 6.2.1):

          An increasing 'slab' memory consumption.

          When running `slabtop` we get a 14g under 'dentry' and counting.

          Show
          moscovig Gilad Moscovitch added a comment - - edited Hey. Do you think it is somehow related to the next issue (using solr 6.2.1): An increasing 'slab' memory consumption. When running `slabtop` we get a 14g under 'dentry' and counting.
          Hide
          markrmiller@gmail.com Mark Miller added a comment -

          Wow this is old. Lost it I guess.

          Show
          markrmiller@gmail.com Mark Miller added a comment - Wow this is old. Lost it I guess.
          Hide
          markrmiller@gmail.com Mark Miller added a comment -

          Hey. Do you think it is somehow related ...

          Unless you are multi-part posting lots of files to Solr (with the tika integration for example), not very likely.

          Show
          markrmiller@gmail.com Mark Miller added a comment - Hey. Do you think it is somehow related ... Unless you are multi-part posting lots of files to Solr (with the tika integration for example), not very likely.
          Hide
          markrmiller@gmail.com Mark Miller added a comment -

          I'll commit this soon.

          Show
          markrmiller@gmail.com Mark Miller added a comment - I'll commit this soon.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit e82399d0677651ad4be1d8d2bdc4777b5d90b0fa in lucene-solr's branch refs/heads/master from markrmiller
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=e82399d ]

          SOLR-1953: It may be possible for temporary files to accumulate until the Solr process is shut down.

          Show
          jira-bot ASF subversion and git services added a comment - Commit e82399d0677651ad4be1d8d2bdc4777b5d90b0fa in lucene-solr's branch refs/heads/master from markrmiller [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=e82399d ] SOLR-1953 : It may be possible for temporary files to accumulate until the Solr process is shut down.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 96b492c05e9384b8dfb9d62823af17cf012fddbb in lucene-solr's branch refs/heads/branch_6x from markrmiller
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=96b492c ]

          SOLR-1953: It may be possible for temporary files to accumulate until the Solr process is shut down.

          Show
          jira-bot ASF subversion and git services added a comment - Commit 96b492c05e9384b8dfb9d62823af17cf012fddbb in lucene-solr's branch refs/heads/branch_6x from markrmiller [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=96b492c ] SOLR-1953 : It may be possible for temporary files to accumulate until the Solr process is shut down.
          Hide
          markrmiller@gmail.com Mark Miller added a comment -

          Thanks Karl! Long lived issue.

          Show
          markrmiller@gmail.com Mark Miller added a comment - Thanks Karl! Long lived issue.
          Hide
          schuch Markus Schuch added a comment - - edited

          We ran into this Issue with an Solr 4.x System and before finding this ticket i found FILEUPLOAD-189. So when someone wants to port the patch to older solr versions, you need also have to look at the commons-fileupload version.

          Show
          schuch Markus Schuch added a comment - - edited We ran into this Issue with an Solr 4.x System and before finding this ticket i found FILEUPLOAD-189 . So when someone wants to port the patch to older solr versions, you need also have to look at the commons-fileupload version.

            People

            • Assignee:
              markrmiller@gmail.com Mark Miller
              Reporter:
              kwright@metacarta.com Karl Wright
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development