Issue Details (XML | Word | Printable)

Key: MODPYTHON-104
Type: New Feature New Feature
Status: Closed Closed
Resolution: Fixed
Priority: Major Major
Assignee: Graham Dumpleton
Reporter: Graham Dumpleton
Votes: 1
Watchers: 0
Operations

If you were logged in you would be able to see more operations.
mod_python

Allow Python code callouts with mod_include (SSI).

Created: 23/Dec/05 09:23 AM   Updated: 05/Apr/07 11:11 AM
Return to search
Component/s: core
Affects Version/s: None
Fix Version/s: 3.3.1

Time Tracking:
Not Specified

File Attachments:
  Size
File Licensed for inclusion in ASF works grahamd_20060126_1_mod_include.diff 2006-01-26 02:44 PM Graham Dumpleton 8 kB
File Licensed for inclusion in ASF works grahamd_20060226_MP104_1.diff 2006-02-26 02:05 PM Graham Dumpleton 11 kB
Text File Licensed for inclusion in ASF works MP104_20060317_jgallacher_1.diff 2006-03-18 01:39 AM Jim Gallacher 0.5 kB

Resolution Date: 27/Oct/06 08:29 AM


 Description  « Hide
The mod_include module supporting server side includes (SSI), provides a means of registering new element tags which trigger callouts to other code in separate Apache modules. This is used for example in mod_perl to allow Perl language code to be used with server side includes:

 <!--#perl sub="MySSI::remote_host" -->

  <!--#perl arg="Hello" arg="SSI" arg="World"
         sub="sub {
                  my($r, @args) = @_;
                  print qq(@args);
              }"
  -->

An equivalent feature for Python was previously asked about on the mailing list back in 2004:

  http://www.modpython.org/pipermail/mod_python/2004-January/014832.html

Since it seems entirely reasonable that such integration of mod_python and mod_include would be possible, thought it would be good to log it as a possible new feature.

Because of SSI's support for basic conditionals, includes and other callout mechanisms, would be a good quick and dirty way of doing templating without having to resort to PSP, or other high level templating systems.

 All   Comments   Work Log   Change History   Subversion Commits      Sort Order: Ascending order - Click to sort in descending order
Nicolas Lehuen made changes - 29/Dec/05 05:03 AM
Field Original Value New Value
Fix Version/s 3.3 [ 12310101 ]
Graham Dumpleton added a comment - 22/Jan/06 06:28 PM
See python-dev mailing list thread:

  http://www.mail-archive.com/python-dev@httpd.apache.org/msg01013.html

for possible syntax and issues. Hopefully feedback will be forthcoming from that. :-)

Graham Dumpleton added a comment - 26/Jan/06 02:44 PM
Attached "grahamd_20060126_1_mod_include.diff".

This includes patch for adding SSI support for Python.

Note that line numbers will probably not match subversion head at this time as I have a lot of other code changes in my local mod_python version. Thus, have generated context diffs, and you will need to look out at how code fits into surrounding code.

What the patch implements is the ability to do stuff like the following in files for which the builtin "INCLUDES" output filter of Apache is enabled. Typically this is for .shtml files but it can be enabled explicitly for other files as well.

<!--#python exec="
from mod_python import apache
import cgi
import sys
parts = apache.import_module('parts')
def _escape(object):
    return cgi.escape(str(object))
"-->
<html>
  <body>
    <pre>
<!--#python eval="_escape(str(globals().keys()))"-->
<!--#python eval="_escape(str(locals().keys()))"-->
<!--#python exec="
print >> filter
for key in filter.req.subprocess_env:
    print >> filter, _escape((key, filter.req.subprocess_env[key]))
"-->
<!--#python eval="parts.content()"-->
    </pre>
  </body>
</html>

One can use either "eval" or "exec". If using "exec", Python code can't have leading indent for first line. Ie., code must be left justified, with indents only used for blocks of code where appropriate. Variables created or modules imported in any code section are accessible to later code sections in that page. Ie., globals/locals are preserved across code sections.

Note that the changes don't do anything special as far as escaping content which may be included in sections of HTML, or in URLs or as values to attributes of HTML elements. It is expected that user do this appropriate for context.

Note also that nothing special is done about Unicode strings. One might consider doing what mod_python.publisher does in 3.2.6 for Unicode strings, but that can only be done for result of an "eval".

In line with how CGI "exec" and mod_perl work, Python code will not be run if "IncludesNOEXEC" option is set.

Graham Dumpleton made changes - 26/Jan/06 02:44 PM
Attachment grahamd_20060126_1_mod_include.diff [ 12322371 ]
Graham Dumpleton made changes - 26/Feb/06 11:23 AM
Assignee Graham Dumpleton [ grahamd ]
Graham Dumpleton made changes - 26/Feb/06 01:57 PM
Status Open [ 1 ] In Progress [ 3 ]
Graham Dumpleton added a comment - 26/Feb/06 02:05 PM
Attached "grahamd_20060226_MP104_1.diff" which is more up to date patch against head of subversion trunk at this time.

A whole new chapter on server side includes possibly needs to be added to documentation as topic doesn't seem to fit under any existing chapter. Will do documentation as sideline activity and not let it stop this patch being pushed into repository so that it might start to be used by interested parties to test it further.

Graham Dumpleton made changes - 26/Feb/06 02:05 PM
Attachment grahamd_20060226_MP104_1.diff [ 12323417 ]
Repository Revision Date User Message
ASF #385257 Sun Mar 12 08:15:05 UTC 2006 grahamd Support added for using Python in content being passed through "INCLUDES"
output filter, or as more commonly referred to server side include (SSI)
mechanism. (MODPYTHON-104)
Files Changed
MODIFY /httpd/mod_python/trunk/test/httpdconf.py
MODIFY /httpd/mod_python/trunk/test/test.py
MODIFY /httpd/mod_python/trunk/src/mod_python.c
MODIFY /httpd/mod_python/trunk/lib/python/mod_python/__init__.py
MODIFY /httpd/mod_python/trunk/Doc/appendixc.tex
MODIFY /httpd/mod_python/trunk/lib/python/mod_python/apache.py
MODIFY /httpd/mod_python/trunk/src/include/mpversion.h
ADD /httpd/mod_python/trunk/test/htdocs/ssi.shtml

Repository Revision Date User Message
ASF #385992 Wed Mar 15 07:48:48 UTC 2006 grahamd Fix SSI suppport so that it compiles and works on Apache 2.2. Fix issue
in SSI where Python code still executed in false part of conditional.
Also remove calls to release interpreter being done prior to interpreter
having been acquired by SSI code in the first place. (MODPYTHON-104)
Files Changed
MODIFY /httpd/mod_python/trunk/src/mod_python.c
MODIFY /httpd/mod_python/trunk/lib/python/mod_python/__init__.py
MODIFY /httpd/mod_python/trunk/src/include/mpversion.h

Repository Revision Date User Message
ASF #386003 Wed Mar 15 08:48:53 UTC 2006 grahamd Use official means for determing when API changed for a specific feature,
namely the AP_MODULE_MAGIC_AT_LEAST macro. (MODPYTHON-104).
Files Changed
MODIFY /httpd/mod_python/trunk/src/include/mod_python.h
MODIFY /httpd/mod_python/trunk/src/mod_python.c
MODIFY /httpd/mod_python/trunk/src/include/mod_python.h.in

Graham Dumpleton added a comment - 15/Mar/06 01:22 PM
Patches for SSI support will not work with Apache 2.2. Names of macros have changed which mean it will not compile and the prototype of a registered handler for include tags has changed meaning that it will then crash when called, even after macros fixed, as arguments don't match up. :-(

Repository Revision Date User Message
ASF #386511 Fri Mar 17 02:39:24 UTC 2006 grahamd Allow a handler to preload data into SSI global data set for #python tag prior
to INCLUDES output filter being executed to process response, by setting the
req.ssi_globals attribute to a dictionary. Also remove filter object explicitly
from global data set after each tag instances is executed in case it causes
issues with garbage collector since request object will reference global data
resulting a cycle. (MODPYTHON-104)
Files Changed
MODIFY /httpd/mod_python/trunk/test/htdocs/tests.py
MODIFY /httpd/mod_python/trunk/test/test.py
MODIFY /httpd/mod_python/trunk/lib/python/mod_python/__init__.py
MODIFY /httpd/mod_python/trunk/lib/python/mod_python/apache.py
MODIFY /httpd/mod_python/trunk/src/include/mpversion.h
MODIFY /httpd/mod_python/trunk/test/htdocs/ssi.shtml

Jim Gallacher added a comment - 17/Mar/06 09:42 PM
I don't have time to review the code changes right now, but code committed up to r386511 is causing a problem on apache 2.0.55 + python 2.3.

Compiler warning:
mod_python.c:1724: warning: implicit declaration of function 'SSI_CREATE_ERROR_BUCKET'

Apache fails to start, with the following error message:
Cannot load /usr/lib/apache2/modules/mod_python.so into server: /usr/lib/apache2/modules/mod_python.so: undefined symbol: SSI_CREATE_ERROR_BUCKET

There is no problem for apache 2.2.0 + python 2.4.2, where it compiles OK and all unit tests pass.

Nicolas Lehuen added a comment - 18/Mar/06 12:17 AM
Same problem with SSI_CREATE_ERROR_BUCKET on Win32, but at compile time.

Deron Meranda added a comment - 18/Mar/06 12:59 AM
This C macro is only defined in mod_include.h for httpd 2.2. In 2.0, it is defined in mod_include.c only.
However, the definition is identical in both 2.0 and 2.2, which is:

#define SSI_CREATE_ERROR_BUCKET(ctx, f, bb) APR_BRIGADE_INSERT_TAIL((bb), \
    apr_bucket_pool_create(apr_pstrdup((ctx)->pool, (ctx)->error_str), \
                           strlen((ctx)->error_str), (ctx)->pool, \
                           (f)->c->bucket_alloc))

Jim Gallacher added a comment - 18/Mar/06 01:38 AM
Comparing the current version with grahamd_20060226_MP104_1.diff, it looks like some of the apache 2.2 code got copied over into the section for apache 2.0. Fix is attached as MP104_20060317_jgallacher_1.diff.

- SSI_CREATE_ERROR_BUCKET(ctx, f, bb);
+ CREATE_ERROR_BUCKET(ctx, tmp_buck, head_ptr, *inserted_head);
+ release_interpreter();

I still haven't reviewed the rest of the code, but unit tests passed once the patch is applied.

Jim Gallacher added a comment - 18/Mar/06 01:39 AM
Fixes SSI_CREATE_ERROR_BUCKET problem for apache 2.0

Jim Gallacher made changes - 18/Mar/06 01:39 AM
Attachment MP104_20060317_jgallacher_1.diff [ 12324298 ]
Repository Revision Date User Message
ASF #386807 Sat Mar 18 04:30:07 UTC 2006 grahamd SSI code specific to Apache 2.0 was incorrectly referencing the Apache 2.1+
macro SSI_CREATE_ERROR_BUCKET. (MODPYTHON-104)
Files Changed
MODIFY /httpd/mod_python/trunk/src/mod_python.c

Repository Revision Date User Message
ASF #396250 Sun Apr 23 10:22:48 UTC 2006 grahamd Error path in SSI code when interpreter could not be obtained was trying
to wrongly release the interpreter. (MODPYTHON-104)
Files Changed
MODIFY /httpd/mod_python/trunk/src/mod_python.c

Graham Dumpleton added a comment - 25/Apr/06 02:02 PM
On Apache 2.2, but not on Apache 2.0, if an error occurs in the Python script, Apache is later crashing at some point.

[Tue Apr 25 16:32:13 2006] [error] [client ::1] includes /Users/grahamd/Workspaces/testing/ssi-1/ssi.shtml: Traceback (most recent call last):
[Tue Apr 25 16:32:13 2006] [error] [client ::1] includes /Users/grahamd/Workspaces/testing/ssi-1/ssi.shtml: File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/site-packages/mod_python/importer.py", line 1377, in IncludeDispatch\n exec(code, filter.req.ssi_globals)
[Tue Apr 25 16:32:13 2006] [error] [client ::1] includes /Users/grahamd/Workspaces/testing/ssi-1/ssi.shtml: File "<string>", line 2, in ?
[Tue Apr 25 16:32:13 2006] [error] [client ::1] includes /Users/grahamd/Workspaces/testing/ssi-1/ssi.shtml: ImportError: No module named module_1
[Tue Apr 25 16:32:14 2006] [error] [client ::1] File does not exist: /usr/local/apache-2.2/htdocs/favicon.ico
[Tue Apr 25 16:32:15 2006] [notice] child pid 5360 exit signal Bus error (10)

Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_PROTECTION_FAILURE at address: 0x00000005
[Switching to process 5368 thread 0x1203]
apr_pool_clear (pool=0x1) at memory/unix/apr_pools.c:685
685 while (pool->child)
(gdb) where
#0 apr_pool_clear (pool=0x1) at memory/unix/apr_pools.c:685
#1 0x00028c4c in includes_filter (f=0x19117f8, b=0x1911998) at mod_include.c:3402
#2 0x00008af8 in default_handler (r=0x190c450) at core.c:3701
#3 0x0000a224 in ap_run_handler (r=0x190c450) at config.c:157
#4 0x0000a7ec in ap_invoke_handler (r=0x190c450) at config.c:371
#5 0x0002e44c in ap_process_request (r=0x190c450) at http_request.c:258
#6 0x0002dac0 in ap_process_http_connection (c=0x1902bd0) at http_core.c:172
#7 0x0001cd64 in ap_run_process_connection (c=0x1902bd0) at connection.c:43
#8 0x000441bc in worker_thread (thd=0x1899920, dummy=0x0) at worker.c:531
#9 0x9002ba68 in _pthread_body ()

The ctx pointer seems to be somehow be getting overwritten with 0.

This may be related to issue Nicolas saw on Win32 with Apache 2.2 in test suite, but then that would imply the Python code in the test failed for some reason which it shouldn't have.

Repository Revision Date User Message
ASF #405027 Mon May 08 12:04:14 UTC 2006 grahamd Eliminate Apache crash after a Python exception in SSI code. Problem was that
filter.disable() was being called, but that expects that f->ctx is an instance
of python_filter_ctx when for SSI it is an include_ctx. As a result it was
overwriting data it shouldn't have. The changes also stop any details of the
exception being written back to the client even if PythonDebug is On as for
SSI it should possibly only ever be logged and errors should not be rendered
in pages. (MODPYTHON-104)
Files Changed
MODIFY /httpd/mod_python/trunk/lib/python/mod_python/importer.py
MODIFY /httpd/mod_python/trunk/lib/python/mod_python/apache.py

Graham Dumpleton added a comment - 08/May/06 06:17 PM
Note that using PythonInterpPerDirective with SSI support would cause a crash. This though has been fixed as part of similar problems when using dynamically register filters as described in MODPYTHON-103.

Graham Dumpleton added a comment - 08/May/06 07:11 PM
The Apache crash when a Python exception occurs has been eliminated, but at the same time have disabled details of any Python exception being rendered into any page sent as result to client even if PythonDebug is On. The rendered page will show the message:

  [an error occurred while processing this directive]

which seems to be more consistent with how SSI handlers work. If found that mod_perl as a comparison, does in some cases render details of exception into page, then maybe this can be changed back at some point.

Note that although the crash has been eliminated, it can still be triggered if filter.disable() is called from Python code executed from SSI code somehow. There perhaps need to be some flag maintained in mod_python filterobject so that the context in which the filter_rec wrapper is used is known and for filter.disable() to only do something when used for an input/output filter and not in other use cases. Other functions of filter object need to be reviewed to see if they might cause other problems if used from an include filter.

Repository Revision Date User Message
ASF #405274 Tue May 09 02:07:52 UTC 2006 grahamd Python code in SSI files can have CRLF line endings on Win32 platform. These
need to be eliminated else Python raises an exception. (MODPYTHON-104)
Files Changed
MODIFY /httpd/mod_python/trunk/lib/python/mod_python/importer.py
MODIFY /httpd/mod_python/trunk/lib/python/mod_python/__init__.py
MODIFY /httpd/mod_python/trunk/lib/python/mod_python/apache.py
MODIFY /httpd/mod_python/trunk/src/include/mpversion.h
MODIFY /httpd/mod_python/trunk/test/htdocs/ssi.shtml

Repository Revision Date User Message
ASF #468288 Fri Oct 27 06:02:43 UTC 2006 grahamd (MODPYTHON-104) Contributed personal article on using Python code with SSI
under mod_python. The original article was sourced from
'http://www.dscpl.com.au/wiki/ModPython/Articles/BasicsOfServerSideIncludes'.
Files Changed
MODIFY /httpd/mod_python/trunk/Doc/modpython6.tex

Graham Dumpleton added a comment - 27/Oct/06 08:29 AM
New chapter added in documentation, so all done.

Graham Dumpleton made changes - 27/Oct/06 08:29 AM
Resolution Fixed [ 1 ]
Status In Progress [ 3 ] Resolved [ 5 ]
Graham Dumpleton made changes - 05/Apr/07 11:11 AM
Status Resolved [ 5 ] Closed [ 6 ]