Uploaded image for project: 'mod_python'
  1. mod_python
  2. MODPYTHON-172

Memory leak with util.fieldstorage using mod_python 3.2.8 on apache 2.0.55

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.2.8
    • Fix Version/s: 3.2.10, 3.3.1
    • Component/s: core
    • Labels:
      None
    • Environment:
      Win32 XP SP1 / SP2
      Apache 2.0.55 installed from binary (.MSI)
      Python 2.4.2 or 2.4.3 installed from binary from www.python.org

      Description

      I encounter memory leaks [~ 16 K per request) using the configuration described below.

      =============================
      Python configuration from Httpd.conf:
      =============================
      Alias /python/ "d:/python24/B2B/"
      <Directory "d:/python24/B2B">
      AddHandler mod_python .py
      PythonHandler pyHandlerHTTP
      PythonDebug On
      </Directory>
      =============================
      Test handler - pyHandlerHTTP.py :
      =============================
      import mod_python
      from mod_python import util

      def handler(req):
      #Removing this line solves the problem.
      F=util.FieldStorage( req )
      return mod_python.apache.OK
      =============================
      HTTP Request (dump using TCPWATCH):
      =============================
      POST http://localhost:80/python/Alertes.py HTTP/1.0
      Content-Type: multipart/form-data; boundary=--------061006144341906
      Content-Length: 209
      Proxy-Connection: keep-alive
      Host: www.tx2-localhost
      Accept: text/html, /
      User-Agent: Mozilla/3.0 (compatible; Indy Library)
      Proxy-Authorization: Basic Og==

      ----------061006144341906
      Content-Disposition: form-data; name="TYPE"

      LAST_ALERTS
      ----------061006144341906
      Content-Disposition: form-data; name="FILEAGE"

      180

      ----------061006144341906

        Activity

        Hide
        haroldship Harold Ship added a comment -

        I've been able to reproduce the problem with "mod_python.publisher".

        Using:
        Windows 2000 Professional
        apache 2.0.54
        python 2.3.5
        mod_python 3.1.3

        the following script:
        #foo.py
        def bar(req):
        a=req.form.getfirst('a')
        req.write('a=%s'%str(a))
        req.write('Ok')

        repeatedly sending requests to <addr>/foo/bar?a=b causes memory to rise continuously. removing the querystring does not. I used task manager to measure memory.

        I've also found that the following change to util.py corrects this behaviour:

        old util.py:
        parse_qs = _apache.parse_qs
        parse_qsl = _apache.parse_qsl

        workaround util.py:
        parse_qs = _apache.parse_qs
        #parse_qsl = _apache.parse_qsl
        from cgi import parse_qsl

        Show
        haroldship Harold Ship added a comment - I've been able to reproduce the problem with "mod_python.publisher". Using: Windows 2000 Professional apache 2.0.54 python 2.3.5 mod_python 3.1.3 the following script: #foo.py def bar(req): a=req.form.getfirst('a') req.write('a=%s'%str(a)) req.write('Ok') repeatedly sending requests to <addr>/foo/bar?a=b causes memory to rise continuously. removing the querystring does not. I used task manager to measure memory. I've also found that the following change to util.py corrects this behaviour: old util.py: parse_qs = _apache.parse_qs parse_qsl = _apache.parse_qsl workaround util.py: parse_qs = _apache.parse_qs #parse_qsl = _apache.parse_qsl from cgi import parse_qsl
        Hide
        haroldship Harold Ship added a comment -

        Excuse me if this is a dumb question, but I don't know anything about the Python-C interface.

        Is it possible that the code in line 319 of _apachemodule.c (from 3.2.8) is the problem?

        The code is:

        if (key && val)
        PyList_Append(pairs, Py_BuildValue("(O,O)", key, val));

        Does the object returned by Py_BuildValue() need to be dereferenced? Something like:

        if (key && val)

        { PyObject * list_elem = Py_BuildValue("(O,O)", key, val); if (list_elem) PyList_Append(pairs, list_elem); Py_XDECREF(list_elem); }
        Show
        haroldship Harold Ship added a comment - Excuse me if this is a dumb question, but I don't know anything about the Python-C interface. Is it possible that the code in line 319 of _apachemodule.c (from 3.2.8) is the problem? The code is: if (key && val) PyList_Append(pairs, Py_BuildValue("(O,O)", key, val)); Does the object returned by Py_BuildValue() need to be dereferenced? Something like: if (key && val) { PyObject * list_elem = Py_BuildValue("(O,O)", key, val); if (list_elem) PyList_Append(pairs, list_elem); Py_XDECREF(list_elem); }
        Hide
        haroldship Harold Ship added a comment -

        Please look at lines 202-205 of the same file, in parse_qs() :

        PyObject *list;
        list = Py_BuildValue("[O]", val);
        PyDict_SetItem(dict, key, list);
        Py_DECREF(list);

        Show
        haroldship Harold Ship added a comment - Please look at lines 202-205 of the same file, in parse_qs() : PyObject *list; list = Py_BuildValue(" [O] ", val); PyDict_SetItem(dict, key, list); Py_DECREF(list);
        Hide
        nlehuen Nicolas Lehuen added a comment -

        Fixed in the trunk, we need to apply changes from revision #420275 if we want to backport this into the 3.2 branch.

        Show
        nlehuen Nicolas Lehuen added a comment - Fixed in the trunk, we need to apply changes from revision #420275 if we want to backport this into the 3.2 branch.
        Hide
        nlehuen Nicolas Lehuen added a comment -

        I've just backported the fix into the 3.2 branch.

        Show
        nlehuen Nicolas Lehuen added a comment - I've just backported the fix into the 3.2 branch.
        Hide
        grahamd Graham Dumpleton added a comment -

        [[ Old comment, sent by email on Sat, 8 Jul 2006 10:27:48 +1000 ]]

        On first look I would agree there is possibly a problem. There also
        would appear to be
        similar issues in other parts of mod_python. For example in
        cfgtree_walk() of util.c it has:

        PyObject *t = Py_BuildValue("(s, s)", dir->directive, dir-
        >args);
        if (!t)
        return PyErr_NoMemory();

        PyList_Append(list, t);

        with "t" not being DECREF'd either.

        Also, in req_readlines() of requestobject.c, we have:

        line = req_readline(self, rlargs);
        while (line && (PyString_Size(line)>0))

        { PyList_Append(result, line); size += PyString_Size(line); if ((sizehint != -1) && (size >= size)) break; line = req_readline(self, args); }

        No DECREF for item added to list.

        I can't see any others in relation to dictionaries yet, but we
        probably need to do a good
        audit of all the code for such things as I have only check
        PyList_Append() and
        PyDict_SetItem() and not the other ways stuff can be added to such
        data structures.

        Anyway, still need to run some tests yet to confirm memory leak.

        Graham

        Show
        grahamd Graham Dumpleton added a comment - [[ Old comment, sent by email on Sat, 8 Jul 2006 10:27:48 +1000 ]] On first look I would agree there is possibly a problem. There also would appear to be similar issues in other parts of mod_python. For example in cfgtree_walk() of util.c it has: PyObject *t = Py_BuildValue("(s, s)", dir->directive, dir- >args); if (!t) return PyErr_NoMemory(); PyList_Append(list, t); with "t" not being DECREF'd either. Also, in req_readlines() of requestobject.c, we have: line = req_readline(self, rlargs); while (line && (PyString_Size(line)>0)) { PyList_Append(result, line); size += PyString_Size(line); if ((sizehint != -1) && (size >= size)) break; line = req_readline(self, args); } No DECREF for item added to list. I can't see any others in relation to dictionaries yet, but we probably need to do a good audit of all the code for such things as I have only check PyList_Append() and PyDict_SetItem() and not the other ways stuff can be added to such data structures. Anyway, still need to run some tests yet to confirm memory leak. Graham
        Hide
        lblanquet Laurent Blanquet added a comment -

        [[ Old comment, sent by email on Fri, 30 Jun 2006 15:03:22 +0200 ]]

        Hello,

        I've tested with Handler :

        import mod_python
        from mod_python import util
        def handler(req):
        #Removing this line solves the problem.
        F=util.FieldStorage( req )
        return mod_python.apache.OK

        And I always see some memory leaks after having installed the patch :

        Laurent.

        ----- Original Message -----
        From: "Harold Ship (JIRA)" <jira@apache.org>
        To: <python-dev@httpd.apache.org>
        Sent: Friday, June 30, 2006 8:58 AM
        Subject: [jira] Commented: (MODPYTHON-172) Memory leak with
        util.fieldstorage using mod_python 3.2.8 on apache 2.0.55

        http://issues.apache.org/jira/browse/MODPYTHON-172?page=comments#action_12418577 ]
        continuously. removing the querystring does not. I used task manager to
        measure memory.
        behaviour:
        2.0.55
        described below.

        Show
        lblanquet Laurent Blanquet added a comment - [[ Old comment, sent by email on Fri, 30 Jun 2006 15:03:22 +0200 ]] Hello, I've tested with Handler : import mod_python from mod_python import util def handler(req): #Removing this line solves the problem. F=util.FieldStorage( req ) return mod_python.apache.OK And I always see some memory leaks after having installed the patch : Laurent. – ----- Original Message ----- From: "Harold Ship (JIRA)" <jira@apache.org> To: <python-dev@httpd.apache.org> Sent: Friday, June 30, 2006 8:58 AM Subject: [jira] Commented: ( MODPYTHON-172 ) Memory leak with util.fieldstorage using mod_python 3.2.8 on apache 2.0.55 http://issues.apache.org/jira/browse/MODPYTHON-172?page=comments#action_12418577 ] continuously. removing the querystring does not. I used task manager to measure memory. behaviour: 2.0.55 described below.
        Hide
        jpg@jgassociates.ca Jim Gallacher added a comment -

        [[ Old comment, sent by email on Fri, 07 Jul 2006 21:26:01 -0400 ]]

        I've been working through the PyList_Append instances and noticed the
        cfgtree_walk and req_readlines issues as well. I've confirmed that
        cfgtree_walk does indeed leak. I haven't tested req_readlines but I
        can't see why it wouldn't leak.

        Can you spot the other bug in req_readlines? Hint: (size >= size) will
        always be true. Once that is fixed I think we need to alter the docs
        for req.readlines() as I'm not sure the description for sizehint
        accurately reflects the way it works.

        Jim

        Show
        jpg@jgassociates.ca Jim Gallacher added a comment - [[ Old comment, sent by email on Fri, 07 Jul 2006 21:26:01 -0400 ]] I've been working through the PyList_Append instances and noticed the cfgtree_walk and req_readlines issues as well. I've confirmed that cfgtree_walk does indeed leak. I haven't tested req_readlines but I can't see why it wouldn't leak. Can you spot the other bug in req_readlines? Hint: (size >= size) will always be true. Once that is fixed I think we need to alter the docs for req.readlines() as I'm not sure the description for sizehint accurately reflects the way it works. Jim

          People

          • Assignee:
            Unassigned
            Reporter:
            lblanquet Laurent Blanquet
          • Votes:
            1 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development