Memory leak with util.fieldstorage using mod_python 3.2.8 on apache 2.0.55

• Key: MODPYTHON-172
• Type: Bug
• Status: Closed
• Resolution: Fixed
• Priority: Major
• Assignee: Unassigned
• Reporter: Laurent Blanquet
• Component/s: core
• Affects Version/s: 3.2.8
• Fix Version/s: 3.2.10, 3.3.1

Environment

Win32 XP SP1 / SP2
Apache 2.0.55 installed from binary (.MSI)
Python 2.4.2 or 2.4.3 installed from binary from www.python.org

Description

I encounter memory leaks [~ 16 K per request) using the configuration described below.

=============================
Python configuration from Httpd.conf:
=============================
Alias /python/ "d:/python24/B2B/"
<Directory "d:/python24/B2B">
        AddHandler mod_python .py
        PythonHandler pyHandlerHTTP
        PythonDebug On
</Directory>
=============================
Test handler - pyHandlerHTTP.py :
=============================
import mod_python
from mod_python import util

def handler(req):
      #Removing this line solves the problem.
      F=util.FieldStorage( req )
      return mod_python.apache.OK
=============================
HTTP Request (dump using TCPWATCH):
=============================
POST http://localhost:80/python/Alertes.py HTTP/1.0
Content-Type: multipart/form-data; boundary=--------061006144341906
Content-Length: 209
Proxy-Connection: keep-alive
Host: www.tx2-localhost
Accept: text/html, */*
User-Agent: Mozilla/3.0 (compatible; Indy Library)
Proxy-Authorization: Basic Og==
 
----------061006144341906
Content-Disposition: form-data; name="TYPE"
 
LAST_ALERTS
----------061006144341906
Content-Disposition: form-data; name="FILEAGE"
 
180
 
----------061006144341906

Comments

[Harold Ship]

I've been able to reproduce the problem with "mod_python.publisher".

Using:
Windows 2000 Professional
apache 2.0.54
python 2.3.5
mod_python 3.1.3

the following script:
#foo.py
def bar(req):
    a=req.form.getfirst('a')
    req.write('a=%s'%str(a))
    req.write('Ok')

repeatedly sending requests to <addr>/foo/bar?a=b causes memory to rise continuously. removing the querystring does not. I used task manager to measure memory.

I've also found that the following change to util.py corrects this behaviour:

old util.py:
parse_qs = _apache.parse_qs
parse_qsl = _apache.parse_qsl


workaround util.py:
parse_qs = _apache.parse_qs
#parse_qsl = _apache.parse_qsl
from cgi import parse_qsl

[Harold Ship]

Excuse me if this is a dumb question, but I don't know anything about the Python-C interface.

Is it possible that the code in line 319 of _apachemodule.c (from 3.2.8) is the problem?

The code is:

            if (key && val)
                PyList_Append(pairs, Py_BuildValue("(O,O)", key, val));

Does the object returned by Py_BuildValue() need to be dereferenced? Something like:

            if (key && val) {
                PyObject * list_elem = Py_BuildValue("(O,O)", key, val);
                if (list_elem)
                    PyList_Append(pairs, list_elem);
                Py_XDECREF(list_elem);
            }

[Harold Ship]

Please look at lines 202-205 of the same file, in parse_qs() :

                    PyObject *list;
                    list = Py_BuildValue("[O]", val);
                    PyDict_SetItem(dict, key, list);
                    Py_DECREF(list);

[Nicolas Lehuen]

Fixed in the trunk, we need to apply changes from revision #420275 if we want to backport this into the 3.2 branch.

[Nicolas Lehuen]

I've just backported the fix into the 3.2 branch.

[Laurent Blanquet]

   [[ Old comment, sent by email on Fri, 30 Jun 2006 15:03:22 +0200 ]]

Hello,



I've tested with Handler :

 import mod_python
from mod_python import util
def handler(req):
       #Removing this line solves the problem.
       F=util.FieldStorage( req )
       return mod_python.apache.OK

And I always see some memory leaks after having installed the patch :


Laurent.

--

----- Original Message -----
From: "Harold Ship (JIRA)" <[e-mail]>
To: <[e-mail]>
Sent: Friday, June 30, 2006 8:58 AM
Subject: [jira] Commented: (MODPYTHON-172) Memory leak with
util.fieldstorage using mod_python 3.2.8 on apache 2.0.55


http://issues.apache.org/jira/browse/MODPYTHON-172?page=comments#action_12418577 ]
continuously. removing the querystring does not. I used task manager to
measure memory.
behaviour:
2.0.55
described below.

[Graham Dumpleton]

   [[ Old comment, sent by email on Sat, 8 Jul 2006 10:27:48 +1000 ]]

On first look I would agree there is possibly a problem. There also
would appear to be
similar issues in other parts of mod_python. For example in
cfgtree_walk() of util.c it has:

         PyObject *t = Py_BuildValue("(s, s)", dir->directive, dir-
 >args);
         if (!t)
             return PyErr_NoMemory();

         PyList_Append(list, t);

with "t" not being DECREF'd either.

Also, in req_readlines() of requestobject.c, we have:

     line = req_readline(self, rlargs);
     while (line && (PyString_Size(line)>0)) {
         PyList_Append(result, line);
         size += PyString_Size(line);
         if ((sizehint != -1) && (size >= size))
             break;
         line = req_readline(self, args);
     }

No DECREF for item added to list.

I can't see any others in relation to dictionaries yet, but we
probably need to do a good
audit of all the code for such things as I have only check
PyList_Append() and
PyDict_SetItem() and not the other ways stuff can be added to such
data structures.

Anyway, still need to run some tests yet to confirm memory leak. :-)

Graham

[Jim Gallacher]

   [[ Old comment, sent by email on Fri, 07 Jul 2006 21:26:01 -0400 ]]


I've been working through the PyList_Append instances and noticed the
cfgtree_walk and req_readlines issues as well. I've confirmed that
cfgtree_walk does indeed leak. I haven't tested req_readlines but I
can't see why it wouldn't leak.

Can you spot the other bug in req_readlines? Hint: (size >= size) will
always be true. ;) Once that is fixed I think we need to alter the docs
for req.readlines() as I'm not sure the description for sizehint
accurately reflects the way it works.

Jim