Uploaded image for project: 'CouchDB'
  1. CouchDB
  2. COUCHDB-1171

Multiple requests to _changes feed causes {error, system_limit} "Too many processes"

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.0.2, 1.0.3, 1.1
    • 1.0.3, 1.1, 1.2
    • None
    • None
    • New Contributors Level (Easy)

    Description

      Originally I have investigated of issue 182 of couchdb-python package where calling db.changes() function over 32768 times generates next messages in CouchDB log:

      [Thu, 19 May 2011 14:03:26 GMT] [info] [<0.2909.0>] 127.0.0.1 - - 'GET' /test/_changes 200
      [Thu, 19 May 2011 14:03:26 GMT] [error] [emulator] Too many processes
      [Thu, 19 May 2011 14:03:26 GMT] [error] [<0.2909.0>] Uncaught error in HTTP request:

      {error,system_limit}
      [Thu, 19 May 2011 14:03:26 GMT] [info] [<0.2909.0>] Stacktrace: [{erlang,spawn, [erlang,apply, [#Fun<couch_stats_collector.1.123391259>,[]]]},
      {erlang,spawn,1},
      {couch_httpd_db,handle_changes_req,2},
      {couch_httpd_db,do_db_req,2},
      {couch_httpd,handle_request_int,5},
      {mochiweb_http,headers,5},
      {proc_lib,init_p_do_apply,3}]
      [Thu, 19 May 2011 14:03:26 GMT] [info] [<0.2909.0>] 127.0.0.1 - - 'GET' /test/_changes 500

      More info about this issue could be found there: http://code.google.com/p/couchdb-python/issues/detail?id=182

      However, I still couldn't reproduce this error using only httplib module, but I've got that same behavior using feed=longpool option:

      from httplib import HTTPConnection
      def test2():
      conn = HTTPConnection('localhost:5984')
      conn.connect()
      i = 0
      while(True):
      conn.putrequest('GET', '/test/_changes?feed=longpool')
      conn.endheaders()
      conn.getresponse().read()
      i = i + 1
      if i % 100 == 0:
      print i

      When i get's around 32667 exception raises
      Traceback (most recent call last):
      File "/home/kxepal/projects/couchdb-python/issue-182/test.py", line 259, in <module>
      test2()
      File "/home/kxepal/projects/couchdb-python/issue-182/test.py", line 239, in test2
      resp.read()
      File "/usr/lib/python2.6/httplib.py", line 522, in read
      return self._read_chunked(amt)
      File "/usr/lib/python2.6/httplib.py", line 565, in _read_chunked
      raise IncompleteRead(''.join(value))
      httplib.IncompleteRead: IncompleteRead(0 bytes read)

      [Thu, 19 May 2011 14:10:20 GMT] [info] [<0.3240.4>] 127.0.0.1 - - 'GET' /test/_changes?feed=longpool 200
      [Thu, 19 May 2011 14:10:20 GMT] [error] [emulator] Too many processes
      [Thu, 19 May 2011 14:10:20 GMT] [error] [<0.3240.4>] Uncaught error in HTTP request: {error,system_limit}

      [Thu, 19 May 2011 14:10:20 GMT] [info] [<0.3240.4>] Stacktrace: [

      {erlang,spawn, [erlang,apply, [#Fun<couch_stats_collector.1.123391259>,[]]]}

      ,

      {erlang,spawn,1}

      ,

      {couch_httpd_db,handle_changes_req,2}

      ,

      {couch_httpd_db,do_db_req,2}

      ,

      {couch_httpd,handle_request_int,5}

      ,

      {mochiweb_http,headers,5}

      ,

      {proc_lib,init_p_do_apply,3}

      ]
      [Thu, 19 May 2011 14:10:20 GMT] [info] [<0.3240.4>] 127.0.0.1 - - 'GET' /test/_changes?feed=longpool 500

      Same error. I know, that test function is quite outside from real use case, but is this correct behavior and couldn't it be used in malicious aims?
      This exception occurres only for multiple requests within single connection for changes feed, chunked lists or attachments are not affected, if I've done all right.

      Test environment:
      Gentoo Linux 2.6.38
      CouchDB 1.0.2 release
      couchdb-python@63feefd9e3b6
      Python 2.6.6

      If there is needed some additional information I could try to provide it.

      Attachments

        1. couchdb-changes-stats-process-leak.patch
          0.7 kB
          Matt Goodall
        2. couchdb-changes-stats-process-leak-test.js
          0.4 kB
          Matt Goodall

        Activity

          People

            rnewson Robert Newson
            kxepal Alexander Shorin
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment