Uploaded image for project: 'mod_python'
  1. mod_python
  2. MODPYTHON-104

Allow Python code callouts with mod_include (SSI).

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.3.1
    • Component/s: core
    • Labels:
      None

      Description

      The mod_include module supporting server side includes (SSI), provides a means of registering new element tags which trigger callouts to other code in separate Apache modules. This is used for example in mod_perl to allow Perl language code to be used with server side includes:

      <!--#perl sub="MySSI::remote_host" -->

      <!--#perl arg="Hello" arg="SSI" arg="World"
      sub="sub

      { my($r, @args) = @_; print qq(@args); }

      "
      -->

      An equivalent feature for Python was previously asked about on the mailing list back in 2004:

      http://www.modpython.org/pipermail/mod_python/2004-January/014832.html

      Since it seems entirely reasonable that such integration of mod_python and mod_include would be possible, thought it would be good to log it as a possible new feature.

      Because of SSI's support for basic conditionals, includes and other callout mechanisms, would be a good quick and dirty way of doing templating without having to resort to PSP, or other high level templating systems.

      1. grahamd_20060126_1_mod_include.diff
        8 kB
        Graham Dumpleton
      2. grahamd_20060226_MP104_1.diff
        11 kB
        Graham Dumpleton
      3. MP104_20060317_jgallacher_1.diff
        0.5 kB
        Jim Gallacher

        Activity

        Hide
        grahamd Graham Dumpleton added a comment -

        See python-dev mailing list thread:

        http://www.mail-archive.com/python-dev@httpd.apache.org/msg01013.html

        for possible syntax and issues. Hopefully feedback will be forthcoming from that.

        Show
        grahamd Graham Dumpleton added a comment - See python-dev mailing list thread: http://www.mail-archive.com/python-dev@httpd.apache.org/msg01013.html for possible syntax and issues. Hopefully feedback will be forthcoming from that.
        Hide
        grahamd Graham Dumpleton added a comment -

        Attached "grahamd_20060126_1_mod_include.diff".

        This includes patch for adding SSI support for Python.

        Note that line numbers will probably not match subversion head at this time as I have a lot of other code changes in my local mod_python version. Thus, have generated context diffs, and you will need to look out at how code fits into surrounding code.

        What the patch implements is the ability to do stuff like the following in files for which the builtin "INCLUDES" output filter of Apache is enabled. Typically this is for .shtml files but it can be enabled explicitly for other files as well.

        <!--#python exec="
        from mod_python import apache
        import cgi
        import sys
        parts = apache.import_module('parts')
        def _escape(object):
        return cgi.escape(str(object))
        "-->
        <html>
        <body>
        <pre>
        <!-#python eval="_escape(str(globals().keys()))"->
        <!-#python eval="_escape(str(locals().keys()))"->
        <!--#python exec="
        print >> filter
        for key in filter.req.subprocess_env:
        print >> filter, _escape((key, filter.req.subprocess_env[key]))
        "-->
        <!-#python eval="parts.content()"->
        </pre>
        </body>
        </html>

        One can use either "eval" or "exec". If using "exec", Python code can't have leading indent for first line. Ie., code must be left justified, with indents only used for blocks of code where appropriate. Variables created or modules imported in any code section are accessible to later code sections in that page. Ie., globals/locals are preserved across code sections.

        Note that the changes don't do anything special as far as escaping content which may be included in sections of HTML, or in URLs or as values to attributes of HTML elements. It is expected that user do this appropriate for context.

        Note also that nothing special is done about Unicode strings. One might consider doing what mod_python.publisher does in 3.2.6 for Unicode strings, but that can only be done for result of an "eval".

        In line with how CGI "exec" and mod_perl work, Python code will not be run if "IncludesNOEXEC" option is set.

        Show
        grahamd Graham Dumpleton added a comment - Attached "grahamd_20060126_1_mod_include.diff". This includes patch for adding SSI support for Python. Note that line numbers will probably not match subversion head at this time as I have a lot of other code changes in my local mod_python version. Thus, have generated context diffs, and you will need to look out at how code fits into surrounding code. What the patch implements is the ability to do stuff like the following in files for which the builtin "INCLUDES" output filter of Apache is enabled. Typically this is for .shtml files but it can be enabled explicitly for other files as well. <!--#python exec=" from mod_python import apache import cgi import sys parts = apache.import_module('parts') def _escape(object): return cgi.escape(str(object)) "--> <html> <body> <pre> <!- #python eval="_escape(str(globals().keys()))" -> <!- #python eval="_escape(str(locals().keys()))" -> <!--#python exec=" print >> filter for key in filter.req.subprocess_env: print >> filter, _escape((key, filter.req.subprocess_env [key] )) "--> <!- #python eval="parts.content()" -> </pre> </body> </html> One can use either "eval" or "exec". If using "exec", Python code can't have leading indent for first line. Ie., code must be left justified, with indents only used for blocks of code where appropriate. Variables created or modules imported in any code section are accessible to later code sections in that page. Ie., globals/locals are preserved across code sections. Note that the changes don't do anything special as far as escaping content which may be included in sections of HTML, or in URLs or as values to attributes of HTML elements. It is expected that user do this appropriate for context. Note also that nothing special is done about Unicode strings. One might consider doing what mod_python.publisher does in 3.2.6 for Unicode strings, but that can only be done for result of an "eval". In line with how CGI "exec" and mod_perl work, Python code will not be run if "IncludesNOEXEC" option is set.
        Hide
        grahamd Graham Dumpleton added a comment -

        Attached "grahamd_20060226_MP104_1.diff" which is more up to date patch against head of subversion trunk at this time.

        A whole new chapter on server side includes possibly needs to be added to documentation as topic doesn't seem to fit under any existing chapter. Will do documentation as sideline activity and not let it stop this patch being pushed into repository so that it might start to be used by interested parties to test it further.

        Show
        grahamd Graham Dumpleton added a comment - Attached "grahamd_20060226_MP104_1.diff" which is more up to date patch against head of subversion trunk at this time. A whole new chapter on server side includes possibly needs to be added to documentation as topic doesn't seem to fit under any existing chapter. Will do documentation as sideline activity and not let it stop this patch being pushed into repository so that it might start to be used by interested parties to test it further.
        Hide
        grahamd Graham Dumpleton added a comment -

        Patches for SSI support will not work with Apache 2.2. Names of macros have changed which mean it will not compile and the prototype of a registered handler for include tags has changed meaning that it will then crash when called, even after macros fixed, as arguments don't match up.

        Show
        grahamd Graham Dumpleton added a comment - Patches for SSI support will not work with Apache 2.2. Names of macros have changed which mean it will not compile and the prototype of a registered handler for include tags has changed meaning that it will then crash when called, even after macros fixed, as arguments don't match up.
        Hide
        jgallacher Jim Gallacher added a comment -

        I don't have time to review the code changes right now, but code committed up to r386511 is causing a problem on apache 2.0.55 + python 2.3.

        Compiler warning:
        mod_python.c:1724: warning: implicit declaration of function 'SSI_CREATE_ERROR_BUCKET'

        Apache fails to start, with the following error message:
        Cannot load /usr/lib/apache2/modules/mod_python.so into server: /usr/lib/apache2/modules/mod_python.so: undefined symbol: SSI_CREATE_ERROR_BUCKET

        There is no problem for apache 2.2.0 + python 2.4.2, where it compiles OK and all unit tests pass.

        Show
        jgallacher Jim Gallacher added a comment - I don't have time to review the code changes right now, but code committed up to r386511 is causing a problem on apache 2.0.55 + python 2.3. Compiler warning: mod_python.c:1724: warning: implicit declaration of function 'SSI_CREATE_ERROR_BUCKET' Apache fails to start, with the following error message: Cannot load /usr/lib/apache2/modules/mod_python.so into server: /usr/lib/apache2/modules/mod_python.so: undefined symbol: SSI_CREATE_ERROR_BUCKET There is no problem for apache 2.2.0 + python 2.4.2, where it compiles OK and all unit tests pass.
        Hide
        nlehuen Nicolas Lehuen added a comment -

        Same problem with SSI_CREATE_ERROR_BUCKET on Win32, but at compile time.

        Show
        nlehuen Nicolas Lehuen added a comment - Same problem with SSI_CREATE_ERROR_BUCKET on Win32, but at compile time.
        Hide
        dmeranda Deron Meranda added a comment -

        This C macro is only defined in mod_include.h for httpd 2.2. In 2.0, it is defined in mod_include.c only.
        However, the definition is identical in both 2.0 and 2.2, which is:

        #define SSI_CREATE_ERROR_BUCKET(ctx, f, bb) APR_BRIGADE_INSERT_TAIL((bb), \
        apr_bucket_pool_create(apr_pstrdup((ctx)>pool, (ctx)>error_str), \
        strlen((ctx)>error_str), (ctx)>pool, \
        (f)>c>bucket_alloc))

        Show
        dmeranda Deron Meranda added a comment - This C macro is only defined in mod_include.h for httpd 2.2. In 2.0, it is defined in mod_include.c only. However, the definition is identical in both 2.0 and 2.2, which is: #define SSI_CREATE_ERROR_BUCKET(ctx, f, bb) APR_BRIGADE_INSERT_TAIL((bb), \ apr_bucket_pool_create(apr_pstrdup((ctx) >pool, (ctx) >error_str), \ strlen((ctx) >error_str), (ctx) >pool, \ (f) >c >bucket_alloc))
        Hide
        jgallacher Jim Gallacher added a comment -

        Comparing the current version with grahamd_20060226_MP104_1.diff, it looks like some of the apache 2.2 code got copied over into the section for apache 2.0. Fix is attached as MP104_20060317_jgallacher_1.diff.

        • SSI_CREATE_ERROR_BUCKET(ctx, f, bb);
          + CREATE_ERROR_BUCKET(ctx, tmp_buck, head_ptr, *inserted_head);
          + release_interpreter();

        I still haven't reviewed the rest of the code, but unit tests passed once the patch is applied.

        Show
        jgallacher Jim Gallacher added a comment - Comparing the current version with grahamd_20060226_MP104_1.diff, it looks like some of the apache 2.2 code got copied over into the section for apache 2.0. Fix is attached as MP104_20060317_jgallacher_1.diff. SSI_CREATE_ERROR_BUCKET(ctx, f, bb); + CREATE_ERROR_BUCKET(ctx, tmp_buck, head_ptr, *inserted_head); + release_interpreter(); I still haven't reviewed the rest of the code, but unit tests passed once the patch is applied.
        Hide
        jgallacher Jim Gallacher added a comment -

        Fixes SSI_CREATE_ERROR_BUCKET problem for apache 2.0

        Show
        jgallacher Jim Gallacher added a comment - Fixes SSI_CREATE_ERROR_BUCKET problem for apache 2.0
        Hide
        grahamd Graham Dumpleton added a comment -

        On Apache 2.2, but not on Apache 2.0, if an error occurs in the Python script, Apache is later crashing at some point.

        [Tue Apr 25 16:32:13 2006] [error] [client ::1] includes /Users/grahamd/Workspaces/testing/ssi-1/ssi.shtml: Traceback (most recent call last):
        [Tue Apr 25 16:32:13 2006] [error] [client ::1] includes /Users/grahamd/Workspaces/testing/ssi-1/ssi.shtml: File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/site-packages/mod_python/importer.py", line 1377, in IncludeDispatch\n exec(code, filter.req.ssi_globals)
        [Tue Apr 25 16:32:13 2006] [error] [client ::1] includes /Users/grahamd/Workspaces/testing/ssi-1/ssi.shtml: File "<string>", line 2, in ?
        [Tue Apr 25 16:32:13 2006] [error] [client ::1] includes /Users/grahamd/Workspaces/testing/ssi-1/ssi.shtml: ImportError: No module named module_1
        [Tue Apr 25 16:32:14 2006] [error] [client ::1] File does not exist: /usr/local/apache-2.2/htdocs/favicon.ico
        [Tue Apr 25 16:32:15 2006] [notice] child pid 5360 exit signal Bus error (10)

        Program received signal EXC_BAD_ACCESS, Could not access memory.
        Reason: KERN_PROTECTION_FAILURE at address: 0x00000005
        [Switching to process 5368 thread 0x1203]
        apr_pool_clear (pool=0x1) at memory/unix/apr_pools.c:685
        685 while (pool->child)
        (gdb) where
        #0 apr_pool_clear (pool=0x1) at memory/unix/apr_pools.c:685
        #1 0x00028c4c in includes_filter (f=0x19117f8, b=0x1911998) at mod_include.c:3402
        #2 0x00008af8 in default_handler (r=0x190c450) at core.c:3701
        #3 0x0000a224 in ap_run_handler (r=0x190c450) at config.c:157
        #4 0x0000a7ec in ap_invoke_handler (r=0x190c450) at config.c:371
        #5 0x0002e44c in ap_process_request (r=0x190c450) at http_request.c:258
        #6 0x0002dac0 in ap_process_http_connection (c=0x1902bd0) at http_core.c:172
        #7 0x0001cd64 in ap_run_process_connection (c=0x1902bd0) at connection.c:43
        #8 0x000441bc in worker_thread (thd=0x1899920, dummy=0x0) at worker.c:531
        #9 0x9002ba68 in _pthread_body ()

        The ctx pointer seems to be somehow be getting overwritten with 0.

        This may be related to issue Nicolas saw on Win32 with Apache 2.2 in test suite, but then that would imply the Python code in the test failed for some reason which it shouldn't have.

        Show
        grahamd Graham Dumpleton added a comment - On Apache 2.2, but not on Apache 2.0, if an error occurs in the Python script, Apache is later crashing at some point. [Tue Apr 25 16:32:13 2006] [error] [client ::1] includes /Users/grahamd/Workspaces/testing/ssi-1/ssi.shtml: Traceback (most recent call last): [Tue Apr 25 16:32:13 2006] [error] [client ::1] includes /Users/grahamd/Workspaces/testing/ssi-1/ssi.shtml: File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/site-packages/mod_python/importer.py", line 1377, in IncludeDispatch\n exec(code, filter.req.ssi_globals) [Tue Apr 25 16:32:13 2006] [error] [client ::1] includes /Users/grahamd/Workspaces/testing/ssi-1/ssi.shtml: File "<string>", line 2, in ? [Tue Apr 25 16:32:13 2006] [error] [client ::1] includes /Users/grahamd/Workspaces/testing/ssi-1/ssi.shtml: ImportError: No module named module_1 [Tue Apr 25 16:32:14 2006] [error] [client ::1] File does not exist: /usr/local/apache-2.2/htdocs/favicon.ico [Tue Apr 25 16:32:15 2006] [notice] child pid 5360 exit signal Bus error (10) Program received signal EXC_BAD_ACCESS, Could not access memory. Reason: KERN_PROTECTION_FAILURE at address: 0x00000005 [Switching to process 5368 thread 0x1203] apr_pool_clear (pool=0x1) at memory/unix/apr_pools.c:685 685 while (pool->child) (gdb) where #0 apr_pool_clear (pool=0x1) at memory/unix/apr_pools.c:685 #1 0x00028c4c in includes_filter (f=0x19117f8, b=0x1911998) at mod_include.c:3402 #2 0x00008af8 in default_handler (r=0x190c450) at core.c:3701 #3 0x0000a224 in ap_run_handler (r=0x190c450) at config.c:157 #4 0x0000a7ec in ap_invoke_handler (r=0x190c450) at config.c:371 #5 0x0002e44c in ap_process_request (r=0x190c450) at http_request.c:258 #6 0x0002dac0 in ap_process_http_connection (c=0x1902bd0) at http_core.c:172 #7 0x0001cd64 in ap_run_process_connection (c=0x1902bd0) at connection.c:43 #8 0x000441bc in worker_thread (thd=0x1899920, dummy=0x0) at worker.c:531 #9 0x9002ba68 in _pthread_body () The ctx pointer seems to be somehow be getting overwritten with 0. This may be related to issue Nicolas saw on Win32 with Apache 2.2 in test suite, but then that would imply the Python code in the test failed for some reason which it shouldn't have.
        Hide
        grahamd Graham Dumpleton added a comment -

        Note that using PythonInterpPerDirective with SSI support would cause a crash. This though has been fixed as part of similar problems when using dynamically register filters as described in MODPYTHON-103.

        Show
        grahamd Graham Dumpleton added a comment - Note that using PythonInterpPerDirective with SSI support would cause a crash. This though has been fixed as part of similar problems when using dynamically register filters as described in MODPYTHON-103 .
        Hide
        grahamd Graham Dumpleton added a comment -

        The Apache crash when a Python exception occurs has been eliminated, but at the same time have disabled details of any Python exception being rendered into any page sent as result to client even if PythonDebug is On. The rendered page will show the message:

        [an error occurred while processing this directive]

        which seems to be more consistent with how SSI handlers work. If found that mod_perl as a comparison, does in some cases render details of exception into page, then maybe this can be changed back at some point.

        Note that although the crash has been eliminated, it can still be triggered if filter.disable() is called from Python code executed from SSI code somehow. There perhaps need to be some flag maintained in mod_python filterobject so that the context in which the filter_rec wrapper is used is known and for filter.disable() to only do something when used for an input/output filter and not in other use cases. Other functions of filter object need to be reviewed to see if they might cause other problems if used from an include filter.

        Show
        grahamd Graham Dumpleton added a comment - The Apache crash when a Python exception occurs has been eliminated, but at the same time have disabled details of any Python exception being rendered into any page sent as result to client even if PythonDebug is On. The rendered page will show the message: [an error occurred while processing this directive] which seems to be more consistent with how SSI handlers work. If found that mod_perl as a comparison, does in some cases render details of exception into page, then maybe this can be changed back at some point. Note that although the crash has been eliminated, it can still be triggered if filter.disable() is called from Python code executed from SSI code somehow. There perhaps need to be some flag maintained in mod_python filterobject so that the context in which the filter_rec wrapper is used is known and for filter.disable() to only do something when used for an input/output filter and not in other use cases. Other functions of filter object need to be reviewed to see if they might cause other problems if used from an include filter.
        Hide
        grahamd Graham Dumpleton added a comment -

        New chapter added in documentation, so all done.

        Show
        grahamd Graham Dumpleton added a comment - New chapter added in documentation, so all done.

          People

          • Assignee:
            grahamd Graham Dumpleton
            Reporter:
            grahamd Graham Dumpleton
          • Votes:
            1 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development