Solr
  1. Solr
  2. SOLR-173

"too many open files" with posting to update handler

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.2
    • Fix Version/s: 1.2
    • Component/s: update
    • Labels:
      None

      Description

      From brian:

      1) Download trunk/nightly
      2) Change line 347 of example/solr/conf/solrconfig.xml to <requestHandler name="/update" class="solr.XmlUpdateRequestHandler">
      3) java -jar start.jar...
      3) Run post.sh a bunch of times on the same xml file... (in a shell script or whatever)
      4) After a few seconds/minutes jetty will crash with "too many open files"

      • - - - -

      all you've got to do is

      apache-solr-nightly/example/exampledocs ryan$ while [ 0 -lt 1 ]; do ./post.sh hd.xml; done

      with the request handler pointing to /update. Use

      1. lsof | grep solr | wc -l

      to watch the fdescs fly.

      1. SOLR-173-open-files-bug.patch
        5 kB
        Ryan McKinley
      2. SOLR-173-open-files-bug.patch
        2 kB
        Ryan McKinley
      3. SOLR-173-open-files-bug.patch
        3 kB
        Ryan McKinley

        Activity

        Hide
        Ryan McKinley added a comment -

        This is not a real 'fix' but it does make the problem go away. I don't understand how or why. Essentially if you use the QueryResponseWriter it runs out of open files - if you write the output directly, it is fine.

        It does not seem to make any difference what is in the response.

        Anyone have any idea what could be going on?

        • - - -

        to make post.sh work with the update handler stuff, you will need to add the content type to <commit/>

        curl $URL --data-binary '<commit/>' -H 'Content-type:text/xml; charset=utf-8'

        Show
        Ryan McKinley added a comment - This is not a real 'fix' but it does make the problem go away. I don't understand how or why. Essentially if you use the QueryResponseWriter it runs out of open files - if you write the output directly, it is fine. It does not seem to make any difference what is in the response. Anyone have any idea what could be going on? - - - to make post.sh work with the update handler stuff, you will need to add the content type to <commit/> curl $URL --data-binary '<commit/>' -H 'Content-type:text/xml; charset=utf-8'
        Hide
        Bertrand Delacretaz added a comment -

        After making the change shown above in solrconfig.xml, a commit is enough to make the number of open file handles grow:

        curl http://localhost:8983/solr/update --data-binary '<commit/>' -H 'Content-type:text/xml; charset=utf-8'

        lsof shows that many of these file handles point to files in data/index, which have been deleted during the commit:

        lsof -p 9563 | grep data/index | wc -l (where 9563 is my solr's process ID)

        shows 398 file handles after a few commits, although my data/index dir contains only 47 files.

        So it looks like something is keeping useless open handles to "old" index files after a commit.

        Show
        Bertrand Delacretaz added a comment - After making the change shown above in solrconfig.xml, a commit is enough to make the number of open file handles grow: curl http://localhost:8983/solr/update --data-binary '<commit/>' -H 'Content-type:text/xml; charset=utf-8' lsof shows that many of these file handles point to files in data/index, which have been deleted during the commit: lsof -p 9563 | grep data/index | wc -l (where 9563 is my solr's process ID) shows 398 file handles after a few commits, although my data/index dir contains only 47 files. So it looks like something is keeping useless open handles to "old" index files after a commit.
        Hide
        Yonik Seeley added a comment -

        The problem is the use of the response writers for something they weren't originally indended for.

        Requests are pretty much always associated with an index searcher. The searcher is grabbed on-demand and used for the remainder of the request's lifetime. This is important (lucene docids can change, etc). At the end of the lifetime of a request, close() must be called to decrement the reference count and close the searcher if necessary. An open searcher holds the files of the index open (so it's view of the index never changes).

        Problem #1: it doesn't look like doFilter() closes the request object.
        Problem #2: update handlers should not request a searcher in the first place... the response writers need to be modified to call getSearcher() on-demand.

        Show
        Yonik Seeley added a comment - The problem is the use of the response writers for something they weren't originally indended for. Requests are pretty much always associated with an index searcher. The searcher is grabbed on-demand and used for the remainder of the request's lifetime. This is important (lucene docids can change, etc). At the end of the lifetime of a request, close() must be called to decrement the reference count and close the searcher if necessary. An open searcher holds the files of the index open (so it's view of the index never changes). Problem #1: it doesn't look like doFilter() closes the request object. Problem #2: update handlers should not request a searcher in the first place... the response writers need to be modified to call getSearcher() on-demand.
        Hide
        Ryan McKinley added a comment -

        this patch takes care of Problem #1, it does not touch #2

        closing the request object keeps the open files from stacking up.

        thanks Yonik

        Show
        Ryan McKinley added a comment - this patch takes care of Problem #1, it does not touch #2 closing the request object keeps the open files from stacking up. thanks Yonik
        Hide
        Ryan McKinley added a comment -

        modifying for problem #2 was easier then i expected, it only touches
        JSONResponseWriter.java
        TextResponseWriter.java
        XMLWriter.java

        rather then initalize the searcher in the constructor, they load a local SolrIndexSearcher inside the writeDocList() function

        Show
        Ryan McKinley added a comment - modifying for problem #2 was easier then i expected, it only touches JSONResponseWriter.java TextResponseWriter.java XMLWriter.java rather then initalize the searcher in the constructor, they load a local SolrIndexSearcher inside the writeDocList() function
        Hide
        Hoss Man added a comment -

        this patch changes the public API of XMLWriter in a way that isn't backwards compatible if anyone was using it in their own plugin ... but i'm guessing the number of people doing that is fairly low (if any) and the added safety it provides seems worth while ... doing some sanity testing and then i'll commit.

        Show
        Hoss Man added a comment - this patch changes the public API of XMLWriter in a way that isn't backwards compatible if anyone was using it in their own plugin ... but i'm guessing the number of people doing that is fairly low (if any) and the added safety it provides seems worth while ... doing some sanity testing and then i'll commit.
        Hide
        Hoss Man added a comment -

        commited

        Show
        Hoss Man added a comment - commited
        Hide
        Hoss Man added a comment -

        This bug was modified as part of a bulk update using the criteria...

        • Marked ("Resolved" or "Closed") and "Fixed"
        • Had no "Fix Version" versions
        • Was listed in the CHANGES.txt for 1.2

        The Fix Version for all 39 issues found was set to 1.2, email notification
        was suppressed to prevent excessive email.

        For a list of all the issues modified, search jira comments for this
        (hopefully) unique string: 20080415hossman2

        Show
        Hoss Man added a comment - This bug was modified as part of a bulk update using the criteria... Marked ("Resolved" or "Closed") and "Fixed" Had no "Fix Version" versions Was listed in the CHANGES.txt for 1.2 The Fix Version for all 39 issues found was set to 1.2, email notification was suppressed to prevent excessive email. For a list of all the issues modified, search jira comments for this (hopefully) unique string: 20080415hossman2

          People

          • Assignee:
            Hoss Man
            Reporter:
            Ryan McKinley
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development