mod_python
  1. mod_python
  2. MODPYTHON-77

The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 3.1.4
    • Fix Version/s: 3.3.1
    • Component/s: core
    • Labels:
      None
    • Environment:
      Python >= 2.3

      Description

      The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3 because of the PEP 311 (Simplified Global Interpreter Lock Acquisition for Extensions):
      ...
      Limitations and Exclusions
      This proposal identifies a solution for extension authors with
      complex multi-threaded requirements, but that only require a
      single "PyInterpreterState". There is no attempt to cater for
      extensions that require multiple interpreter states. At the time
      of writing, no extension has been identified that requires
      multiple PyInterpreterStates, and indeed it is not clear if that
      facility works correctly in Python itself.
      ...

      For mod_python this means, that complex Python extensions won't work any more with Python >= 2.3, because they are supposed to work only with the first interpreter state initialized for the current process (a problem we experienced). The first interpreter state is not used by mod_python after the python_init is called.

      One solution, which works fine for me, is to save the first interpreter state into the "interpreters" dictionary in the function python_init (MAIN_INTERPRETER is used as a key):

      static int python_init(apr_pool_t *p, apr_pool_t *ptemp,
      apr_pool_t *plog, server_rec *s)
      {

      ...

      /* initialize global Python interpreter if necessary */
      if (! Py_IsInitialized())
      {

      /* initialze the interpreter */
      Py_Initialize();

      #ifdef WITH_THREAD
      /* create and acquire the interpreter lock */
      PyEval_InitThreads();
      #endif
      /* create the obCallBack dictionary */
      interpreters = PyDict_New();
      if (! interpreters)

      { ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0, s, "python_init: PyDict_New() failed! No more memory?"); exit(1); } { /* Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions BEGIN */ PyObject *p = 0; interpreterdata * idata = (interpreterdata *)malloc(sizeof(interpreterdata)); PyThreadState* currentThreadState = PyThreadState_Get(); PyInterpreterState *istate = currentThreadState->interp; idata->istate = istate; /* obcallback will be created on first use */ idata->obcallback = NULL; p = PyCObject_FromVoidPtr((void ) idata, NULL); /*p->refcout = 1*/ PyDict_SetItemString(interpreters, MAIN_INTERPRETER, p); /*p->refcout = 2*/ Py_DECREF(p); /*p->refcout = 1*/ /* END Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions */ }

      /* Release the thread state because we will never use

      • the main interpreter, only sub interpreters created later. */
        PyThreadState_Swap(NULL);

      #ifdef WITH_THREAD
      /* release the lock; now other threads can run */
      PyEval_ReleaseLock();
      #endif

      }
      return OK;
      }

      Another change I've made in the attached file is to Py_DECREF(p) in get_interpreter, which will remove leaky reference to the PyCObject with the interpreter data. This was not a real problem, but now I see fewer leaks in BoundsChecker .

      1. src.zip
        17 kB
        Boyan Boyadjiev
      2. mod_python.h.diff
        0.5 kB
        Boyan Boyadjiev
      3. mod_python.c.diff
        9 kB
        Boyan Boyadjiev
      4. mod_python.c
        67 kB
        Boyan Boyadjiev
      5. grahamd_20051105.tar.gz
        16 kB
        Graham Dumpleton
      6. gilstate.tar.gz
        1 kB
        Graham Dumpleton
      7. gil_test.c
        2 kB
        Boyan Boyadjiev
      8. diff3.txt
        2 kB
        Boyan Boyadjiev
      9. diff2.txt
        2 kB
        Boyan Boyadjiev
      10. diff.txt
        1 kB
        Boyan Boyadjiev

        Activity

        Graham Dumpleton made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Graham Dumpleton made changes -
        Status In Progress [ 3 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Graham Dumpleton made changes -
        Status Open [ 1 ] In Progress [ 3 ]
        Graham Dumpleton made changes -
        Fix Version/s 3.3 [ 12310101 ]
        Assignee Graham Dumpleton [ grahamd ]
        Graham Dumpleton made changes -
        Attachment grahamd_20051105.tar.gz [ 12320489 ]
        Graham Dumpleton made changes -
        Attachment gilstate.tar.gz [ 12314896 ]
        Boyan Boyadjiev made changes -
        Attachment mod_python.h.diff [ 12312723 ]
        Attachment mod_python.c.diff [ 12312724 ]
        Boyan Boyadjiev made changes -
        Attachment src.zip [ 12312714 ]
        Boyan Boyadjiev made changes -
        Attachment gil_test.c [ 12312713 ]
        Boyan Boyadjiev made changes -
        Attachment diff3.txt [ 12312173 ]
        Boyan Boyadjiev made changes -
        Attachment diff2.txt [ 12312172 ]
        Boyan Boyadjiev made changes -
        Attachment diff.txt [ 12312160 ]
        Boyan Boyadjiev made changes -
        Field Original Value New Value
        Attachment mod_python.c [ 12312154 ]
        Boyan Boyadjiev created issue -

          People

          • Assignee:
            Graham Dumpleton
            Reporter:
            Boyan Boyadjiev
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development