[#MODPYTHON-143] Implement and integrate a new module importer.

mod_python

Implement and integrate a new module importer.

Created: 10/Mar/06 01:28 PM Updated: 11/Apr/07 11:33 AM

Return to search

Component/s:

importer

Affects Version/s:

3.2.8

Fix Version/s:

3.3.1

Time Tracking:

Issue & Sub-Tasks

Issue Only

Not Specified

Issue Links:

dependent

This issue is depended upon by:

~~MODPYTHON-115~~ import_module() and multiple modules of same name.

~~MODPYTHON-116~~ Attributes removed from module code file still accessible until restart.

~~MODPYTHON-8~~ Improve apache.load_module with a two-level locking scheme

~~MODPYTHON-54~~ Add a way to import a published page into another published page

Resolution Date:

06/Dec/06 09:33 AM

Sub-Tasks:

All

Open

No sub-tasks match this view.

Description

« Hide

This is an overall task to cover the issue of rectifying the various module importer issues by replacing it with a new implementation. A description of the various problems can be found in:

  http://www.dscpl.com.au/articles/modpython-003.html

Separate issues had already been created for some of the specific problems. These issues will now be linked to this problem and thus marked as being dependent on this issue.

In other words, replacing the module importer will solve a number of number issues. Rather than try and keep up to date all the separate issues, all information about the replacement will be put against this issue instead.

Note that there are also some issues which are not directly related to the module importer but which will be made dependent on this issue because it is easier to fix the issue as part of the rewrite of the module importer and top level handler dispatch mechanism than it is to address it as a distinct item.

In respect of what impacts the new module importer implementation may have and how it is used may change, this will be documented in the following document for the time being:

  http://www.dscpl.com.au/articles/modpython-007.html

Note that this document is a work in progress. It is dense reading and assumes you know a bit about the current module importer and its problems. Any significant issues raised by this document can be added here as a comment, or if a general dicussion of a topic is needed, raise the issue on the mod_python developers mailing list.

A possible new implementation for the module importer is basically ready for testing and experimentation. The intent is to push it into the mod_python source tree, but for its use to be optional.

If wanting to enable it for a specific Python interpreter, the PythonImport directive would be used:

  PythonImport mod_python.future.importer mytestinterpreter

If wanting to enable it for all Python interpreters, a PythonOption directive would be used at global scope within the Apache configuration. Ie., outside of all Location, Directory or Files container directives. The exact option name to be used hasn't yet been decided.

More details and announcements at the appropriate time.

Description

This is an overall task to cover the issue of rectifying the various module importer issues by replacing it with a new implementation. A description of the various problems can be found in: http://www.dscpl.com.au/articles/modpython-003.html Separate issues had already been created for some of the specific problems. These issues will now be linked to this problem and thus marked as being dependent on this issue. In other words, replacing the module importer will solve a number of number issues. Rather than try and keep up to date all the separate issues, all information about the replacement will be put against this issue instead. Note that there are also some issues which are not directly related to the module importer but which will be made dependent on this issue because it is easier to fix the issue as part of the rewrite of the module importer and top level handler dispatch mechanism than it is to address it as a distinct item. In respect of what impacts the new module importer implementation may have and how it is used may change, this will be documented in the following document for the time being: http://www.dscpl.com.au/articles/modpython-007.html Note that this document is a work in progress. It is dense reading and assumes you know a bit about the current module importer and its problems. Any significant issues raised by this document can be added here as a comment, or if a general dicussion of a topic is needed, raise the issue on the mod_python developers mailing list. A possible new implementation for the module importer is basically ready for testing and experimentation. The intent is to push it into the mod_python source tree, but for its use to be optional. If wanting to enable it for a specific Python interpreter, the PythonImport directive would be used: PythonImport mod_python.future.importer mytestinterpreter If wanting to enable it for all Python interpreters, a PythonOption directive would be used at global scope within the Apache configuration. Ie., outside of all Location, Directory or Files container directives. The exact option name to be used hasn't yet been decided. More details and announcements at the appropriate time.

Show »

All

Comments

Work Log

Change History

Subversion Commits

Sort Order:

Repository	Revision	Date	User	Message
ASF	#387186	Mon Mar 20 11:18:21 UTC 2006	grahamd	Added new experimental module importer. New module importer does not run by default but must be enabled explicitly. (~~MODPYTHON-143~~)
				Files Changed
				MODIFY /httpd/mod_python/trunk/test/htdocs/tests.py MODIFY /httpd/mod_python/trunk/test/test.py ADD /httpd/mod_python/trunk/lib/python/mod_python/importer.py MODIFY /httpd/mod_python/trunk/lib/python/mod_python/__init__.py MODIFY /httpd/mod_python/trunk/lib/python/mod_python/apache.py MODIFY /httpd/mod_python/trunk/src/include/mpversion.h

Repository	Revision	Date	User	Message
ASF	#389413	Tue Mar 28 06:18:06 UTC 2006	grahamd	Can't avoid doing stat on file if module marked as dirty from previously failed import as mtime will never be updated and it will reload on every subsequent request. Code also wasn't distinguishing empty dictionary from None properly, meaning that per request module cache wasn't being used, resulting in potential loss of efficiency in certain cases and danger of loading multiple instances of same module in context of one request if file on disk change at that instance in time. (~~MODPYTHON-143~~)
				Files Changed
				MODIFY /httpd/mod_python/trunk/lib/python/mod_python/importer.py MODIFY /httpd/mod_python/trunk/lib/python/mod_python/__init__.py MODIFY /httpd/mod_python/trunk/src/include/mpversion.h

Repository	Revision	Date	User	Message
ASF	#396248	Sun Apr 23 10:05:43 UTC 2006	grahamd	Remove the "~/" shortcut for referencing modules by path stored in directory specified by the Python*Handler directive, when using apache.import_module() and when specifying the handler as option to the directive. In the case of the option to the directive itself, the "./" and "../" shorcuts can be used instead. (~~MODPYTHON-143~~)
				Files Changed
				MODIFY /httpd/mod_python/trunk/lib/python/mod_python/importer.py

Repository	Revision	Date	User	Message
ASF	#425700	Wed Jul 26 11:04:29 UTC 2006	grahamd	Added 'mp' qualifier into the names of the special mod_python variables put into every module loaded by the new module importer. Changes are __info__ to __mp_info__, __clone__ to __mp_clone__ and __purge__ to __mp_purge__. Code has been restructured so that module specific search path additions can now be appended to module variable called __mp_path__ instead of __info__.path. (~~MODPYTHON-143~~)
				Files Changed
				MODIFY /httpd/mod_python/trunk/lib/python/mod_python/importer.py MODIFY /httpd/mod_python/trunk/lib/python/mod_python/__init__.py MODIFY /httpd/mod_python/trunk/src/include/mpversion.h

Repository	Revision	Date	User	Message
ASF	#432916	Sun Aug 20 01:02:28 UTC 2006	grahamd	(~~MODPYTHON-143~~) Made the new importer the default. Now have to use the PythonOption 'mod_python.legacy.importer' to specify names of interpreters or '*' for all interpreters, if you want to use old importer. Now possible to set PythonOption 'mod_python.importer.path' to specify a search path for modules. Ie., like PythonPath, but only for the new importer. Feature whereby possible to use '~/' as prefix to explicit module name to the apache.import_module() function now reinstated. This allows a way of importing module easily from directory for which the handler directive was specified. The PythonOption 'mod_python.import.search_handler_root' has been dropped as same thing can be achieved using 'mod_python.importer.path' option or by explicit importation from handler root. Added two new functions apache.modules_graph() and apache.request_modules_graph() which return as a string a DOT graph definition of relationship between all modules imported or just those specific to the current request. These functions are only present when new importer is being used.
				Files Changed
				MODIFY /httpd/mod_python/trunk/test/test.py MODIFY /httpd/mod_python/trunk/lib/python/mod_python/importer.py MODIFY /httpd/mod_python/trunk/lib/python/mod_python/__init__.py MODIFY /httpd/mod_python/trunk/lib/python/mod_python/apache.py MODIFY /httpd/mod_python/trunk/src/include/mpversion.h

Repository	Revision	Date	User	Message
ASF	#466349	Sat Oct 21 06:07:34 UTC 2006	grahamd	(~~MODPYTHON-143~~) Ensure that details of exceptions that occur in __mp_clone__() and __mp_purge__() are logged.
				Files Changed
				MODIFY /httpd/mod_python/trunk/lib/python/mod_python/importer.py

Repository	Revision	Date	User	Message
ASF	#466361	Sat Oct 21 07:44:27 UTC 2006	grahamd	(~~MODPYTHON-143~~) Include additional information about process, interpreter etc in the details logged and returned for an exception occuring in a handler.
				Files Changed
				MODIFY /httpd/mod_python/trunk/lib/python/mod_python/importer.py

Repository	Revision	Date	User	Message
ASF	#468630	Sat Oct 28 05:03:34 UTC 2006	grahamd	(~~MODPYTHON-143~~) Dump out information about modules touched by a request when an exception occurs in the importing of the modules or in executing the handler and the exception isn't caught and is resulting in an internal server error response.
				Files Changed
				MODIFY /httpd/mod_python/trunk/lib/python/mod_python/importer.py MODIFY /httpd/mod_python/trunk/lib/python/mod_python/__init__.py MODIFY /httpd/mod_python/trunk/src/include/mpversion.h

Repository	Revision	Date	User	Message
ASF	#468633	Sat Oct 28 06:17:38 UTC 2006	grahamd	(~~MODPYTHON-143~~) Can't set generation to zero prior to import as that stops dependency checks from working properly. The modification time is already set back to zero on failed import which is what should have been checked to indicate failed import.
				Files Changed
				MODIFY /httpd/mod_python/trunk/lib/python/mod_python/importer.py

Repository	Revision	Date	User	Message
ASF	#468634	Sat Oct 28 06:40:41 UTC 2006	grahamd	(~~MODPYTHON-143~~) When cache is frozen or autoreload is disabled and a module had previously failed to import, still need to ensure that an attempt is made to reload it even though autoreload is now turned off. This may still result in inconsistances with multiple instances of a module loaded. Short story is that turning autoreloading off when modules haven't all been loaded successfully in the first place is not good.
				Files Changed
				MODIFY /httpd/mod_python/trunk/lib/python/mod_python/importer.py

Repository	Revision	Date	User	Message
ASF	#468657	Sat Oct 28 07:35:30 UTC 2006	grahamd	(~~MODPYTHON-143~~) Use a separate flag in the cache to indicate that a module must be reloaded on next access. This is instead of setting modification time back to zero. This works out to be better as still can interogate the modification time for debugging purposes.
				Files Changed
				MODIFY /httpd/mod_python/trunk/lib/python/mod_python/importer.py

Repository	Revision	Date	User	Message
ASF	#468856	Sun Oct 29 09:27:27 UTC 2006	grahamd	(~~MODPYTHON-143~~) Allow '~/' shortcut to be used in paths listed in the 'mod_python.importer.path' option. This means that paths can be specified with respect to handler root rather than using absolute paths in Apache configuration files.
				Files Changed
				MODIFY /httpd/mod_python/trunk/lib/python/mod_python/importer.py

Repository	Revision	Date	User	Message
ASF	#469055	Mon Oct 30 03:32:20 UTC 2006	grahamd	(~~MODPYTHON-143~~) Fix module cache debug output whereby it dies when no modules in per request cache. Also update output so it lists children as paths instead of magic module name. Also distinguish friends from children, friends being modules that were imported as part of the execution of the request handler and not module import and therefore aren't included in dependency checks to determine if module should be reloaded.
				Files Changed
				MODIFY /httpd/mod_python/trunk/lib/python/mod_python/importer.py MODIFY /httpd/mod_python/trunk/lib/python/mod_python/__init__.py MODIFY /httpd/mod_python/trunk/src/include/mpversion.h

Repository	Revision	Date	User	Message
ASF	#469065	Mon Oct 30 04:56:11 UTC 2006	grahamd	(~~MODPYTHON-143~~) Cleanup code formatting debug output for children and friends.
				Files Changed
				MODIFY /httpd/mod_python/trunk/lib/python/mod_python/importer.py

Repository	Revision	Date	User	Message
ASF	#472056	Tue Nov 07 10:34:25 UTC 2006	grahamd	(~~MODPYTHON-143~~) Minimal documentation on some, but not all, changes to the apache.import_module() function. Also corrections to documentation of request objects functions for registering handlers and filters.
				Files Changed
				MODIFY /httpd/mod_python/trunk/Doc/modpython4.tex

Repository	Revision	Date	User	Message
ASF	#473171	Fri Nov 10 02:53:27 UTC 2006	grahamd	(~~MODPYTHON-143~~) Added entry to list of changes for implementation of new module importer.
				Files Changed
				MODIFY /httpd/mod_python/trunk/Doc/appendixc.tex

Repository	Revision	Date	User	Message
ASF	#481507	Sat Dec 02 06:18:17 UTC 2006	grahamd	(~~MODPYTHON-143~~) Have the handler root fallback to be the document root when handler directives are used in Location directive or in root context of VirtualHost or actual server. This handler root is what is used as the first place to look for a module when using apache.import_module() or when it is necessary to expand a path prefixed by '~/'.
				Files Changed
				MODIFY /httpd/mod_python/trunk/lib/python/mod_python/importer.py MODIFY /httpd/mod_python/trunk/lib/python/mod_python/__init__.py MODIFY /httpd/mod_python/trunk/src/include/mpversion.h

Repository	Revision	Date	User	Message
ASF	#481741	Sun Dec 03 09:14:06 UTC 2006	grahamd	(~~MODPYTHON-143~~) Allow just '~' to be used as a directory in the module importer search path to refer to the handler root directory.
				Files Changed
				MODIFY /httpd/mod_python/trunk/lib/python/mod_python/importer.py