Created attachment 28475 [details] mod_jk.log showing full run from Apache start through failed HTML page Since October 2011, I have run the mod_jk/1.2.32 connector. I built the connector myself using Apache 2.2.15's apxs (I also built Apache myself). I downloaded, built, and installed the latest mod_jk.1.2.33 connector. This version causes my Apache server to crash when serving most any page I've tried--HTML, cgi-bin/test-cgi, GIF, etc. The only exceptions I've found are Apache's htdocs/index.html page, server-info, and Tomcat served pages. Initially I tried building the connector using --with-apxs set to the apxs in my copy of Apache 2.2.15. When that failed, I downloaded, built, and installed Apache 2.2.22, then built mod_jk 1.2.33 with the apxs from that version. Again, Apache (this time 2.2.22) crashes with mod_jk 1.2.33, but runs fine with the 1.2.32 connector. I am running openSUSE Linux 11.4. From the connector's config.log: uname -m = i686 uname -r = 2.6.37.6-0.11-desktop uname -s = Linux uname -v = #1 SMP PREEMPT 2011-12-19 23:39:38 +0100 To document the problem, I added "CoreDumpDirectory /tmp" to httpd.conf, set "JkLogLevel trace" for mod_jk, made sure I was going to load mod_jk/1.2.33, and started Tomcat and Apache (I don't see Tomcat as necessary for this, but just be thorough). I accessed a simple HTML page. (I get similar results trying to access the local Apache manual, local javadocs, etc.) I'm attaching the mod_jk.log that resulted. Below is the entire contents of the Apache error_log: [Thu Mar 15 09:05:14 2012] [warn] Init: Session Cache is not configured [hint: SSLSessionCache] [Thu Mar 15 09:05:14 2012] [notice] Apache/2.2.22 (Unix) mod_ssl/2.2.22 OpenSSL/1.0.0c mod_jk/1.2.33 configured -- resuming normal operations [Thu Mar 15 09:05:22 2012] [notice] child pid 24552 exit signal Segmentation fault (11), possible coredump in /tmp Here is what I find in /tmp/core with gdb: $ gdb /srv/apache2/bin/httpd /tmp/core GNU gdb (GDB) SUSE (7.2-3.3) Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "i586-suse-linux". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /srv/apache2.2.22/bin/httpd...done. [New Thread 24552] ... Loaded symbols for /lib/ld-linux.so.2 Reading symbols from /srv/apache2/modules/mod_jk.so...done. Loaded symbols for /srv/apache2/modules/mod_jk.so Core was generated by `/srv/apache2.2.22/bin/httpd -k start'. Program terminated with signal 11, Segmentation fault. #0 0xb720288c in jk_map_to_storage () from /srv/apache2/modules/mod_jk.so (gdb) backtrace #0 0xb720288c in jk_map_to_storage () from /srv/apache2/modules/mod_jk.so #1 0x08081478 in ap_run_map_to_storage () #2 0x08081e55 in ap_process_request_internal () #3 0x080da502 in ap_process_request () #4 0x080d7531 in ap_process_http_connection () #5 0x0808f05f in ap_run_process_connection () #6 0x0808f474 in ap_process_connection () #7 0x080f3deb in child_main () #8 0x080f3fe0 in make_child () #9 0x080f403a in startup_children () #10 0x080f44d8 in ap_mpm_run () #11 0x08070e89 in main () (gdb) I can provide the full core dump if required.
Are you able to build mod_jk with symbols, e.g. adding "-g" to your gcc flags? That way gdb would show us the line number where the crash happens. Furthermore the request you used for reproduction is for "/~thad/". Does it also happen if you directly retrieve a mapped URL explicitely, like e.g. an index.jsp or similar, and on the other hand a non mapped object directly, like an index.txt? I will try to reproduce tomorrow. Regards, Rainer
This happens if I directly retrieve a mapped URL explicitly. Mounts made with JkMount work fine. "JkMount /*.jsp ajp13" and "JkMount /*/servlet/* ajp13" allow me to run the Tomcat examples, as do my JkMount's apps. Oddly, HTML and images placed directly under the server root (/srv/apache2.2.22/htdocs) display fine. Those same files placed in another directory will not display though the directory has been Alias'ed (the files display fine if mod_jk/1.2.32 is loaded vs 1.2.33). Apache's manual, Alias'ed as manual, will not display with mod_jk/1.2.33 loaded but will with 1.2.33. I will rebuild mod_jk with the -g option on Monday and provide you the backtrace then. Thanks.
Confirmed on Debian Linux, Apache 2.2.16, mod_jk 1.2.33, using JkMounts and all other settings that have been in place for years in working configuration. Some specifics of my configuration: DocumentRoot /var/www/... Alias /context /path/to/my/context JkMount /context/*.jsp workerX JkMount /context/... Accessing files in DocumentRoot works. Making a request to /context/a.jsp cores every time. Making a request to /context/a.css (not JkMounted) cores every time. Accessing a file in DocumentRoot (again) works. At first, I thought this wasn't working because I was getting connection-resets for everything, but I was not able to reproduce it. I'll re-build with -g and get a stack trace.
Debian isn't making my life easy. I did build with debugging symbols: $ grep CFLAGS Makefile APXSCFLAGS = -DLINUX=2 -D_FORTIFY_SOURCE=2 -D_GNU_SOURCE -D_REENTRANT -I/usr/include/apr-1.0 -I/usr/include/openssl -I/usr/include/xmltok -pthread -DHAVE_APR -I/usr/include/apr-1.0 -I/usr/include/apr-1.0 -g -DHAVE_CONFIG_H CFLAGS = -g -DHAVE_CONFIG_H My old 1.2.23: $ ls -l /usr/lib/apache2/modules/mod_jk-1.2.32.so -rwxr-xr-x 1 root root 483286 Dec 7 11:28 /usr/lib/apache2/modules/mod_jk-1.2.32.so $ file /usr/lib/apache2/modules/mod_jk-1.2.32.so /usr/lib/apache2/modules/mod_jk-1.2.32.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped My new 1.2.33: $ file /usr/lib/apache2/modules/mod_jk-1.2.33.so /usr/lib/apache2/modules/mod_jk-1.2.33.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped $ ls -l /usr/lib/apache2/modules/mod_jk-1.2.33.so -rwxr-xr-x 1 root root 769392 Mar 16 16:35 /usr/lib/apache2/modules/mod_jk-1.2.33.so The new version is nearly twice the size as the older one: I suspect that debugging symbols *have* been compiled-in. Yet: gdb /usr/sbin/apache2 core GNU gdb (GDB) 7.0.1-debian Copyright (C) 2009 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... warning: The current binary is a PIE (Position Independent Executable), which GDB does NOT currently support. Most debugger features will fail if used in this session. Reading symbols from /usr/sbin/apache2...(no debugging symbols found)...done. [New Thread 8888] [New Thread 8899] [New Thread 8900] [New Thread 8901] [New Thread 8903] [New Thread 8905] [New Thread 8892] [New Thread 8907] [New Thread 8894] [New Thread 8909] [New Thread 8896] [New Thread 8912] [New Thread 8914] [New Thread 8916] [New Thread 8918] [New Thread 8920] [New Thread 8934] [New Thread 8922] [New Thread 8895] [New Thread 8893] [New Thread 8924] [New Thread 8891] [New Thread 8926] [New Thread 8932] [New Thread 8928] [New Thread 8930] Core was generated by `/usr/sbin/apache2 -k start'. Program terminated with signal 11, Segmentation fault. #0 0x00007f63638f56aa in ?? () (gdb) where #0 0x00007f63638f56aa in ?? () #1 0x0000000000000030 in ?? () #2 0x00007f63657a6c61 in ?? () #3 0x00007f636954cbc0 in ?? () #4 0x00007f6369982a30 in ?? () #5 0x0000000000000000 in ?? () (gdb) I guess this means that since httpd doesn't have debugging symbols, we can't see anything. Also, it's using DSOs, so those probably don't get loaded by gdb anyway. I'll see what I can do. More environment information: $ uname -a Linux dev.chadis.com 2.6.32-312-ec2 #24-Ubuntu SMP Fri Jan 7 18:30:50 UTC 2011 x86_64 GNU/Linux $ /usr/sbin/apache2 -V Server version: Apache/2.2.16 (Debian) Server built: Feb 5 2012 21:35:44 Server's Module Magic Number: 20051115:24 Server loaded: APR 1.4.2, APR-Util 1.3.9 Compiled using: APR 1.4.2, APR-Util 1.3.9 Architecture: 64-bit Server MPM: Worker threaded: yes (fixed thread count) forked: yes (variable process count) Server compiled with.... -D APACHE_MPM_DIR="server/mpm/worker" -D APR_HAS_SENDFILE -D APR_HAS_MMAP -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled) -D APR_USE_SYSVSEM_SERIALIZE -D APR_USE_PTHREAD_SERIALIZE -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT -D APR_HAS_OTHER_CHILD -D AP_HAVE_RELIABLE_PIPED_LOGS -D DYNAMIC_MODULE_LIMIT=128 -D HTTPD_ROOT="/etc/apache2" -D SUEXEC_BIN="/usr/lib/apache2/suexec" -D DEFAULT_PIDLOG="/var/run/apache2.pid" -D DEFAULT_SCOREBOARD="logs/apache_runtime_status" -D DEFAULT_ERRORLOG="logs/error_log" -D AP_TYPES_CONFIG_FILE="mime.types" -D SERVER_CONFIG_FILE="apache2.conf"
Reviewing diff of current native/apache-2.0/mod_jk.c against r1142157 (aka 1.2.32 tag) That is: [1] http://svn.apache.org/viewvc/tomcat/jk/trunk/native/apache-2.0/mod_jk.c?r1=1240904&r2=1142157&diff_format=h The request-specific configuration structure was changed from (rule_extension_t *) to new structure (jk_request_conf_t *). That was done in r1187906 Looking at the hook methods in mod_jk.c that access jk_request_conf_t: 1. jk_translate Does palloc of new instance unconditionally 2. jk_map_to_storage Checks for NULL and conditionally does palloc of new instance. 3. request_log_transaction Checks for NULL and returns DECLINED if it is null 4. jk_handler and init_ws_service (supplementary method called from jk_handler) Access the structure without checking it for NULL. Those are lines @r1293818: 805, 807; 2575, 2577; 2603, 2605; 2625, 2627 I think that jk_handler method should get the jk_request_conf_t instance once somewhere near the top of the method and check it for being NULL. There are 3 possible ways to react if it is NULL: a) return DECLINED, b) palloc a new instance like jk_map_to_storage does. c) skip the code that accesses the structure + jk_request_conf_t *rconf = ap_get_module_config(r->request_config, &jk_module); + if (rconf == NULL) { + jk_request_conf_t *rconf = apr_palloc(r->pool, sizeof(jk_request_conf_t)); + rconf->jk_handled = JK_FALSE; + rconf->rule_extensions = NULL; + ap_set_module_config(r->request_config, &jk_module, rconf); + } +
So I was able to reproduce it as well. Seems the crash is inside handling rule extensions Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0xab1e7b40 (LWP 4822)] 0xb7cbd3ae in jk_map_to_storage (r=0xaa0024c8) at mod_jk.c:3783 3783 rconf->rule_extensions = e; (gdb) bt #0 0xb7cbd3ae in jk_map_to_storage (r=0xaa0024c8) at mod_jk.c:3783 #1 0x08080598 in ap_run_map_to_storage () #2 0x0808123e in ap_process_request_internal () #3 0x080a2ca3 in ap_process_async_request () #4 0x0809f762 in ap_process_http_async_connection () #5 0x0809f932 in ap_process_http_connection () #6 0x08095e03 in ap_run_process_connection ()
Fixed in r1301987 Suppose 1.2.34 is coming pretty soon :)
(In reply to comment #7) > Fixed in r1301987 > Suppose 1.2.34 is coming pretty soon :) I build trunk and tried it -- no more SIGSEGV :)
I've checked out 1301987, built, and tried it. All okay here with mod_jk/1.2.34-dev