input filter hangs in combination with mod_proxy

• Key: MODPYTHON-76
• Type: Bug
• Status: Open
• Priority: Major
• Assignee: Graham Dumpleton
• Reporter: Martijn Faassen
• Component/s: core
• Affects Version/s: 3.1.4, 3.2.7

Description

Input filters hang when mod_proxy is in use. Other behavior seen is infinite calls of the input filter function even though the data has already
been processed.

In Apache 2.0.24, this problem seems to be fundamental; I have no way to get rid of the hang, except perhaps in the case of an input filter that
does not change the size of the passed-through data.

In svn Apache, the problem goes away as soon as I remove the filter.flush() in apache.py's FilterDispatch handler. svn Apache does according to its changelog contain a fix to mod_proxy's handling of input filters.

For my application, use of mod_proxy is essential, as mod_python is essentially proxying another web server process.

Output filters do not seem to be affected by the presence of filter.flush(), and in fact filter.flush() is important to make sure memory use remains small when handling large amounts of data being outputted by the server.

All of this is rather hairy and I don't know quite what to blame; part of this seems indeed due to Apache itself, but perhaps not everything.

Comments

[Graham Dumpleton]

For further details on this issue see developer mailing list posts.
  
  http://article.gmane.org/gmane.comp.apache.mod-python.devel/1341/match=martijn
  http://article.gmane.org/gmane.comp.apache.mod-python.devel/1350/match=martijn

The offending package that might be used as a test case is called Tramline, which is available from:

  http://faassen.n--tree.net/blog/view/weblog/2005/11/11/0
  http://codespeak.net/svn/rr/tramline/trunk/

If it really is a requirement that automatic flushing needs to be able to be turned off, suggest that add a new configuration directive called "PythonFlushFilter" be implemented. The default if not defined would be "On", with it being able to be specified as "Off".

But then, the mailing list posts suggest that the workaround of disabling the flush may not work for all Apache versions.

[Graham Dumpleton]

I wander if the issue with filter.flush() could be related to the fact that FilterDispatch will call it after the filter function has returned, even if the filter function had closed the filter. For simple examples this doesn't seem to matter, but maybe for Tramline the way it does things means that the flush after close causes a problem.

I can't see any harm in saying:

  if not filter.closed:
    filter.flush()

Anyway, asking originator of bug report to see if it makes a difference or not.

[Graham Dumpleton]

No response from the person who reported this yet, so am trying again.

In the mean time, I have seen myself in my own experiments that an output filter can be called an extra time after the filter has already been closed because of the mandatory flush that is occuring. This doesn't seem to happen in all cases though. The fact that a filter is called after the close has occured is definitely not good though as any state it has may be bogus and thus problems could occur.

Thus reasonably confident that fix above would most likely solve the original problem.

[Graham Dumpleton]

Latest update on this issue is that the change of not flushing if filter closed does not in itself fix the originally reported problem.