req.add_handler() and inheritance of directory to be searched for module

• Key: MODPYTHON-155
• Type: Sub-task
• Status: Closed
• Resolution: Fixed
• Priority: Major
• Assignee: Graham Dumpleton
• Reporter: Graham Dumpleton
• Component/s: importer
• Fix Version/s: 3.3.1

Description

The documentation for req.add_handler() says:

"""Optional dir is a string containing the name of the directory to be added to the pythonpath. If no directory is specified, then, if there is already a handler of the same type specified, its directory is inherited, otherwise the directory of the presently executing handler is used. I there is a PythonPath directive in effect, then sys.path will be set exactly according to it (no directories added, the dir argument is ignored)."""

This comment about the directory being inherited from the prior or currently executing handler is actually bogus as the code does not do anything specific at all to try and implement such behaviour. If it works this way at all it is partly by luck as what will actually dictate where the module specified to the req.add_handler() method is found is the current order of directories specified in sys.path. Since additional directories added into sys.path by the old importer can be performed in effectively random order, behaviour could actually be quite random if the same module name were used in multiple directories.

Because the new importer doesn't add directories into sys.path for Python*Handler directives, a problem will currently arise if no directory is supplied to req.add_handler(). Specifically, a module may not be able to be found. This is because it can no longer fall back on to fact that with old module importer, the directory corresponding to the Python*Handler directive would be listed in sys.path somewhere.

Thus, the documented behaviour for req.add_handler() when the directory hasn't been set needs to actually be implemented as described with an appropriate directory being calculated at the time that req.add_handler() is called with that directory being recorded as needing to be searched for the module. In changing the code though, if old and new importers are going to be supported during a transition phase, it must detect when the new module importer is being used and only do this when it is, as otherwise it will screw up how modules are found for the old importer.

Comments

[Graham Dumpleton]

To fix this, handler list now contains reference back to parent handler that registered it dynamically. This is available as req.hlist.parent. Note that if for some reason a dynamically added handler was in turn added by a dynamically added handler, the immediate parent directory can be None. Thus need to keep tracking back through parents until non None directory found or top handler found. The directory can even then still be None if for example it was invoked out of a Location directive.

A similar list back to parent handlers was also needed for filters which were dynamically registered. This is available using filter.parent.

The new module importer has been updated to use these parent lists and thus now possible to say:

from mod_python import apache

def uppercase(filter):
    s = filter.read()
    while s:
        filter.write(s.upper())
        s = filter.read()
    if s is None:
        filter.close()

def handler(req):
    req.add_output_filter("UPPERCASE")
    req.content_type = 'text/plain'
    req.write('handler')
    return apache.OK

def fixuphandler(req):
    req.handler = 'mod_python'
    req.register_output_filter("UPPERCASE", "handlers::uppercase")
    req.add_handler('PythonHandler', "handlers::handler")
    return apache.OK

and it will work correctly as it would have with old module importer.

[Graham Dumpleton]

Marked this resolved too soon. The code works fine, but looks like it could be memory inefficient due to the MpHList_FromHLEntry() function creating a new Apache memory pool and then also copying the underlying handler list when creating the handler list entry wrapper object. A new one of these handler list entry wrapper objects is going to be created each time parent() is called. Thus sub optimal at present.

[Graham Dumpleton]

Memory inefficiency fixed, but more importantly new problem caused by fixes for memory leaks described in MODPYTHON-181 also addressed.