Bug 52921 - mod_jk 1.2.33 crashes Apache 2.2 when accessing HTML, GIF, etc.
mod_jk 1.2.33 crashes Apache 2.2 when accessing HTML, GIF, etc.
Status: RESOLVED FIXED
Product: Tomcat Connectors
Classification: Unclassified
Component: mod_jk
1.2.33
PC Linux
: P2 major (vote)
: ---
Assigned To: Tomcat Developers Mailing List
:
Depends on:
Blocks:
  Show dependency tree
 
Reported: 2012-03-15 18:05 UTC by Thad Humphries
Modified: 2012-03-19 17:38 UTC (History)
0 users



Attachments
mod_jk.log showing full run from Apache start through failed HTML page (48.37 KB, text/x-log)
2012-03-15 18:05 UTC, Thad Humphries
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Thad Humphries 2012-03-15 18:05:33 UTC
Created attachment 28475 [details]
mod_jk.log showing full run from Apache start through failed HTML page

Since October 2011, I have run the mod_jk/1.2.32 connector. I built the connector myself using Apache 2.2.15's apxs (I also built Apache myself).

I downloaded, built, and installed the latest mod_jk.1.2.33 connector. This version causes my Apache server to crash when serving most any page I've tried--HTML, cgi-bin/test-cgi, GIF, etc. The only exceptions I've found are Apache's htdocs/index.html page, server-info, and Tomcat served pages.

Initially I tried building the connector using --with-apxs set to the apxs in my copy of Apache 2.2.15. When that failed, I downloaded, built, and installed Apache 2.2.22, then built mod_jk 1.2.33 with the apxs from that version. Again, Apache (this time 2.2.22) crashes with mod_jk 1.2.33, but runs fine with the 1.2.32 connector.

I am running openSUSE Linux 11.4. From the connector's config.log:
uname -m = i686
uname -r = 2.6.37.6-0.11-desktop
uname -s = Linux
uname -v = #1 SMP PREEMPT 2011-12-19 23:39:38 +0100

To document the problem, I added "CoreDumpDirectory /tmp" to httpd.conf, set "JkLogLevel trace" for mod_jk, made sure I was going to load mod_jk/1.2.33, and started Tomcat and Apache (I don't see Tomcat as necessary for this, but just be thorough). I accessed a simple HTML page. (I get similar results trying to access the local Apache manual, local javadocs, etc.)

I'm attaching the mod_jk.log that resulted. Below is the entire contents of the Apache error_log:

[Thu Mar 15 09:05:14 2012] [warn] Init: Session Cache is not configured [hint: SSLSessionCache]
[Thu Mar 15 09:05:14 2012] [notice] Apache/2.2.22 (Unix) mod_ssl/2.2.22 OpenSSL/1.0.0c mod_jk/1.2.33 configured -- resuming normal operations
[Thu Mar 15 09:05:22 2012] [notice] child pid 24552 exit signal Segmentation fault (11), possible coredump in /tmp


Here is what I find in /tmp/core with gdb:

$ gdb /srv/apache2/bin/httpd /tmp/core
GNU gdb (GDB) SUSE (7.2-3.3)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i586-suse-linux".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /srv/apache2.2.22/bin/httpd...done.
[New Thread 24552]
...
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /srv/apache2/modules/mod_jk.so...done.
Loaded symbols for /srv/apache2/modules/mod_jk.so
Core was generated by `/srv/apache2.2.22/bin/httpd -k start'.
Program terminated with signal 11, Segmentation fault.
#0  0xb720288c in jk_map_to_storage () from /srv/apache2/modules/mod_jk.so
(gdb) backtrace
#0  0xb720288c in jk_map_to_storage () from /srv/apache2/modules/mod_jk.so
#1  0x08081478 in ap_run_map_to_storage ()
#2  0x08081e55 in ap_process_request_internal ()
#3  0x080da502 in ap_process_request ()
#4  0x080d7531 in ap_process_http_connection ()
#5  0x0808f05f in ap_run_process_connection ()
#6  0x0808f474 in ap_process_connection ()
#7  0x080f3deb in child_main ()
#8  0x080f3fe0 in make_child ()
#9  0x080f403a in startup_children ()
#10 0x080f44d8 in ap_mpm_run ()
#11 0x08070e89 in main ()
(gdb) 

I can provide the full core dump if required.
Comment 1 Rainer Jung 2012-03-15 20:40:13 UTC
Are you able to build mod_jk with symbols, e.g. adding "-g" to your gcc flags? That way gdb would show us the line number where the crash happens.

Furthermore the request you used for reproduction is for "/~thad/". Does it also happen if you directly retrieve a mapped URL explicitely, like e.g. an index.jsp or similar, and on the other hand a non mapped object directly, like an index.txt?

I will try to reproduce tomorrow.

Regards,

Rainer
Comment 2 Thad Humphries 2012-03-15 21:24:04 UTC
This happens if I directly retrieve a mapped URL explicitly.

Mounts made with JkMount work fine. "JkMount /*.jsp ajp13" and "JkMount /*/servlet/* ajp13" allow me to run the Tomcat examples, as do my JkMount's apps.

Oddly, HTML and images placed directly under the server root (/srv/apache2.2.22/htdocs) display fine. Those same files placed in another directory will not display though the directory has been Alias'ed (the files display fine if mod_jk/1.2.32 is loaded vs 1.2.33).

Apache's manual, Alias'ed as manual, will not display with mod_jk/1.2.33 loaded but will with 1.2.33.

I will rebuild mod_jk with the -g option on Monday and provide you the backtrace then.

Thanks.
Comment 3 Christopher Schultz 2012-03-16 20:33:05 UTC
Confirmed on Debian Linux, Apache 2.2.16, mod_jk 1.2.33, using JkMounts and all other settings that have been in place for years in working configuration.

Some specifics of my configuration:

DocumentRoot /var/www/...
Alias /context /path/to/my/context
JkMount /context/*.jsp workerX
JkMount /context/...

Accessing files in DocumentRoot works.
Making a request to /context/a.jsp cores every time.
Making a request to /context/a.css (not JkMounted) cores every time.
Accessing a file in DocumentRoot (again) works. At first, I thought this wasn't working because I was getting connection-resets for everything, but I was not able to reproduce it.

I'll re-build with -g and get a stack trace.
Comment 4 Christopher Schultz 2012-03-16 20:54:18 UTC
Debian isn't making my life easy. I did build with debugging symbols:

$ grep CFLAGS Makefile
APXSCFLAGS = -DLINUX=2 -D_FORTIFY_SOURCE=2 -D_GNU_SOURCE -D_REENTRANT -I/usr/include/apr-1.0 -I/usr/include/openssl -I/usr/include/xmltok -pthread   -DHAVE_APR  -I/usr/include/apr-1.0 -I/usr/include/apr-1.0 -g -DHAVE_CONFIG_H
CFLAGS = -g -DHAVE_CONFIG_H

My old 1.2.23:
$ ls -l /usr/lib/apache2/modules/mod_jk-1.2.32.so 
-rwxr-xr-x 1 root root 483286 Dec  7 11:28 /usr/lib/apache2/modules/mod_jk-1.2.32.so
$ file /usr/lib/apache2/modules/mod_jk-1.2.32.so 
/usr/lib/apache2/modules/mod_jk-1.2.32.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped

My new 1.2.33:
$ file /usr/lib/apache2/modules/mod_jk-1.2.33.so 
/usr/lib/apache2/modules/mod_jk-1.2.33.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
$ ls -l /usr/lib/apache2/modules/mod_jk-1.2.33.so 
-rwxr-xr-x 1 root root 769392 Mar 16 16:35 /usr/lib/apache2/modules/mod_jk-1.2.33.so

The new version is nearly twice the size as the older one: I suspect that debugging symbols *have* been compiled-in.

Yet:

 gdb /usr/sbin/apache2 core
GNU gdb (GDB) 7.0.1-debian
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...

warning: The current binary is a PIE (Position Independent Executable), which
GDB does NOT currently support.  Most debugger features will fail if used
in this session.

Reading symbols from /usr/sbin/apache2...(no debugging symbols found)...done.
[New Thread 8888]
[New Thread 8899]
[New Thread 8900]
[New Thread 8901]
[New Thread 8903]
[New Thread 8905]
[New Thread 8892]
[New Thread 8907]
[New Thread 8894]
[New Thread 8909]
[New Thread 8896]
[New Thread 8912]
[New Thread 8914]
[New Thread 8916]
[New Thread 8918]
[New Thread 8920]
[New Thread 8934]
[New Thread 8922]
[New Thread 8895]
[New Thread 8893]
[New Thread 8924]
[New Thread 8891]
[New Thread 8926]
[New Thread 8932]
[New Thread 8928]
[New Thread 8930]
Core was generated by `/usr/sbin/apache2 -k start'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007f63638f56aa in ?? ()
(gdb) where
#0  0x00007f63638f56aa in ?? ()
#1  0x0000000000000030 in ?? ()
#2  0x00007f63657a6c61 in ?? ()
#3  0x00007f636954cbc0 in ?? ()
#4  0x00007f6369982a30 in ?? ()
#5  0x0000000000000000 in ?? ()
(gdb) 

I guess this means that since httpd doesn't have debugging symbols, we can't see anything. Also, it's using DSOs, so those probably don't get loaded by gdb anyway. I'll see what I can do.

More environment information:
$ uname -a
Linux dev.chadis.com 2.6.32-312-ec2 #24-Ubuntu SMP Fri Jan 7 18:30:50 UTC 2011 x86_64 GNU/Linux


$ /usr/sbin/apache2 -V
Server version: Apache/2.2.16 (Debian)
Server built:   Feb  5 2012 21:35:44
Server's Module Magic Number: 20051115:24
Server loaded:  APR 1.4.2, APR-Util 1.3.9
Compiled using: APR 1.4.2, APR-Util 1.3.9
Architecture:   64-bit
Server MPM:     Worker
  threaded:     yes (fixed thread count)
    forked:     yes (variable process count)
Server compiled with....
 -D APACHE_MPM_DIR="server/mpm/worker"
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=128
 -D HTTPD_ROOT="/etc/apache2"
 -D SUEXEC_BIN="/usr/lib/apache2/suexec"
 -D DEFAULT_PIDLOG="/var/run/apache2.pid"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="mime.types"
 -D SERVER_CONFIG_FILE="apache2.conf"
Comment 5 Konstantin Kolinko 2012-03-16 23:41:19 UTC
Reviewing diff of current native/apache-2.0/mod_jk.c against r1142157 (aka 1.2.32 tag)

That is:
[1] http://svn.apache.org/viewvc/tomcat/jk/trunk/native/apache-2.0/mod_jk.c?r1=1240904&r2=1142157&diff_format=h


The request-specific configuration structure was changed from (rule_extension_t *) to new structure (jk_request_conf_t *). That was done in r1187906


Looking at the hook methods in mod_jk.c that access jk_request_conf_t:

1. jk_translate 

  Does palloc of new instance unconditionally

2. jk_map_to_storage

  Checks for NULL and conditionally does palloc of new instance.

3. request_log_transaction

  Checks for NULL and returns DECLINED if it is null

4. jk_handler
  and init_ws_service (supplementary method called from jk_handler)

  Access the structure without checking it for NULL.
  Those are lines @r1293818:
805, 807; 2575, 2577; 2603, 2605; 2625, 2627

I think that jk_handler method should get the jk_request_conf_t instance once somewhere near the top of the method and check it for being NULL.
There are 3 possible ways to react if it is NULL:

  a) return DECLINED,
  b) palloc a new instance like jk_map_to_storage does.
  c) skip the code that accesses the structure

+    jk_request_conf_t *rconf = ap_get_module_config(r->request_config, &jk_module);
+    if (rconf == NULL) {
+        jk_request_conf_t *rconf = apr_palloc(r->pool, sizeof(jk_request_conf_t));
+        rconf->jk_handled = JK_FALSE;
+        rconf->rule_extensions = NULL;
+        ap_set_module_config(r->request_config, &jk_module, rconf);
+    }
+
Comment 6 Mladen Turk 2012-03-17 19:17:22 UTC
So I was able to reproduce it as well.
Seems the crash is inside handling rule extensions

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xab1e7b40 (LWP 4822)]
0xb7cbd3ae in jk_map_to_storage (r=0xaa0024c8) at mod_jk.c:3783
3783	                rconf->rule_extensions = e;

(gdb) bt
#0  0xb7cbd3ae in jk_map_to_storage (r=0xaa0024c8) at mod_jk.c:3783
#1  0x08080598 in ap_run_map_to_storage ()
#2  0x0808123e in ap_process_request_internal ()
#3  0x080a2ca3 in ap_process_async_request ()
#4  0x0809f762 in ap_process_http_async_connection ()
#5  0x0809f932 in ap_process_http_connection ()
#6  0x08095e03 in ap_run_process_connection ()
Comment 7 Mladen Turk 2012-03-17 20:01:40 UTC
Fixed in r1301987
Suppose 1.2.34 is coming pretty soon :)
Comment 8 Christopher Schultz 2012-03-19 16:18:16 UTC
(In reply to comment #7)
> Fixed in r1301987
> Suppose 1.2.34 is coming pretty soon :)

I build trunk and tried it -- no more SIGSEGV :)
Comment 9 Thad Humphries 2012-03-19 17:38:49 UTC
I've checked out 1301987, built, and tried it. All okay here with mod_jk/1.2.34-dev