Sun ask me to forward this request on to you guys. --------------------- Report --------------------- category : hotspot subcategory : release : 5.0 type : bug synopsis : JVM crashes randomly with tomcat customer name : Daniel Hopper customer mail : dhopper@bzresults.com sdn id : demagogue language : en company : BzResults hardware : x86 os : linux bug id : 0 date created : Fri Aug 26 06:52:39 MDT 2005 date evaluated : Fri Aug 26 11:39:11 MDT 2005 description : FULL PRODUCT VERSION : Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_04-b05) Java HotSpot(TM) Server VM (build 1.5.0_04-b05, mixed mode) FULL OS VERSION : Linux bz-web1.bzproductions.com 2.4.21-27.0.2.ELsmp #1 SMP Wed Jan 12 23:35:44 EST 2005 i686 i686 i386 GNU/Linux EXTRA RELEVANT SYSTEM CONFIGURATION : Running tomcat 5.5.9 with jsvc A DESCRIPTION OF THE PROBLEM : Tomcat server randomly crashes with the error # # An unexpected error has been detected by HotSpot Virtual Machine: # # SIGSEGV (0xb) at pc=0x00b29a6d, pid=29779, tid=84360112 # # Java VM: Java HotSpot(TM) Client VM (1.5.0_04-b05 mixed mode) # Problematic frame: # C [libc.so.6+0x74a6d] # # An error report file with more information is saved as /tmp/hs_err_pid29779.log # # If you would like to submit a bug report, please visit: # http://java.sun.com/webapps/bugreport/crash.jsp # It tends to be completely random THE PROBLEM WAS REPRODUCIBLE WITH -Xint FLAG: Did not try THE PROBLEM WAS REPRODUCIBLE WITH -server FLAG: Did not try STEPS TO FOLLOW TO REPRODUCE THE PROBLEM : Happens at random times. In random segments of code EXPECTED VERSUS ACTUAL BEHAVIOR : Shouldn't crash ERROR MESSAGES/STACK TRACES THAT OCCUR : # # An unexpected error has been detected by HotSpot Virtual Machine: # # SIGSEGV (0xb) at pc=0x00b29a6d, pid=29779, tid=84360112 # # Java VM: Java HotSpot(TM) Client VM (1.5.0_04-b05 mixed mode) # Problematic frame: # C [libc.so.6+0x74a6d] # --------------- T H R E A D --------------- Current thread (0xb1779350): JavaThread "Timer-64" daemon [_thread_in_native, id=2327] siginfo:si_signo=11, si_errno=0, si_code=1, si_addr=0x00000000 Registers: EAX=0x00000000, EBX=0x00be9898, ECX=0x00000000, EDX=0x00000001 ESP=0x0507352c, EBP=0x05073544, ESI=0x00000000, EDI=0x00000000 EIP=0x00b29a6d, CR2=0x00000000, EFLAGS=0x00010202 Top of Stack: (sp=0x0507352c) 0x0507352c: 00000006 00000038 00bea180 00be9898 0x0507353c: 00bea180 00000001 05073560 00b25eb6 0x0507354c: b1779350 050735b0 008c5f38 b177940c 0x0507355c: 00000037 050735b0 008b4569 00000038 0x0507356c: ada0beec b44b92dd b449f9cc b1779350 0x0507357c: 00000000 00000000 b1779350 b1779350 0x0507358c: 00000000 ada0beec 00000000 00000000 0x0507359c: 00000275 008b442b 008c5f38 004452a0 Instructions: (pc=0x00b29a6d) 0x00b29a5d: 74 26 00 89 c8 65 83 3d 0c 00 00 00 00 3e 74 01 0x00b29a6d: f0 0f b1 16 0f 95 c0 84 c0 0f 84 c3 00 00 00 8b Stack: [0x04ff3000,0x05074000), sp=0x0507352c, free space=513k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) C [libc.so.6+0x74a6d] C [libc.so.6+0x70eb6] malloc+0x76 C [libjava.so+0x10569] JNU_GetStringPlatformChars+0x149 C [libjava.so+0x15b7d] Java_java_io_UnixFileSystem_getLastModifiedTime+0x6d J java.io.UnixFileSystem.getLastModifiedTime(Ljava/io/File;)J J java.io.File.lastModified()J j net.bzprod.apec.util.FileMonitor.run()V+91 J java.util.TimerThread.mainLoop()V j java.util.TimerThread.run()V+1 v ~StubRoutines::call_stub V [libjvm.so+0x16bf8c] V [libjvm.so+0x25e5f8] V [libjvm.so+0x16b7e5] V [libjvm.so+0x16b87e] V [libjvm.so+0x1d0fd5] V [libjvm.so+0x2b76d3] V [libjvm.so+0x25f108] C [libpthread.so.0+0x4de8] Java frames: (J=compiled Java code, j=interpreted, Vv=VM code) J java.io.UnixFileSystem.getLastModifiedTime(Ljava/io/File;)J J java.io.File.lastModified()J j net.bzprod.apec.util.FileMonitor.run()V+91 J java.util.TimerThread.mainLoop()V j java.util.TimerThread.run()V+1 v ~StubRoutines::call_stub --------------- P R O C E S S --------------- Java Threads: ( => current thread ) 0xaa424658 JavaThread "TP-Processor32" daemon [_thread_in_native, id=4788] 0xaa47f138 JavaThread "TP-Processor31" daemon [_thread_blocked, id=4787] 0xa87dcec8 JavaThread "TP-Processor30" daemon [_thread_blocked, id=4786] 0xaa427360 JavaThread "TP-Processor29" daemon [_thread_blocked, id=4785] 0xa937b458 JavaThread "TP-Processor28" daemon [_thread_blocked, id=4784] 0xa87fbc28 JavaThread "TP-Processor27" daemon [_thread_in_native, id=4783] 0xaa441608 JavaThread "TP-Processor26" daemon [_thread_blocked, id=4782] 0xa876f950 JavaThread "TP-Processor25" daemon [_thread_blocked, id=4781] 0xaafe8200 JavaThread "Timer-76" daemon [_thread_blocked, id=4370] 0xac7bf6e8 JavaThread "TP-Processor24" daemon [_thread_blocked, id=4280] 0xad966c10 JavaThread "TP-Processor23" daemon [_thread_blocked, id=4279] 0xb16bd3e0 JavaThread "TP-Processor22" daemon [_thread_blocked, id=4278] 0xae3cfb38 JavaThread "TP-Processor21" daemon [_thread_blocked, id=4277] 0xa9383520 JavaThread "Timer-75" daemon [_thread_blocked, id=4269] 0xad4d41b0 JavaThread "Timer-74" daemon [_thread_blocked, id=4139] 0xa8e655d8 JavaThread "Timer-73" daemon [_thread_in_native, id=4123] 0xa852da88 JavaThread "Timer-72" daemon [_thread_blocked, id=4107] 0xaa1f71e0 JavaThread "Timer-71" daemon [_thread_blocked, id=3926] 0xad995f00 JavaThread "TP-Processor20" daemon [_thread_blocked, id=3851] 0xad995200 JavaThread "TP-Processor19" daemon [_thread_in_native, id=3850] 0xaeadc2f8 JavaThread "TP-Processor18" daemon [_thread_in_native, id=3849] 0xac85ece8 JavaThread "TP-Processor17" daemon [_thread_blocked, id=3848] 0xa877f090 JavaThread "Timer-70" daemon [_thread_blocked, id=3847] 0xb0445ee0 JavaThread "TP-Processor16" daemon [_thread_blocked, id=3571] 0xb041d738 JavaThread "TP-Processor15" daemon [_thread_blocked, id=3570] 0xada192c0 JavaThread "TP-Processor14" daemon [_thread_blocked, id=3569] 0xaed03330 JavaThread "TP-Processor13" daemon [_thread_in_native, id=3568] 0xa9395248 JavaThread "Timer-69" daemon [_thread_blocked, id=3567] 0xa9354318 JavaThread "Timer-68" daemon [_thread_blocked, id=2773] 0xab350590 JavaThread "Timer-67" daemon [_thread_blocked, id=2692] 0xaa455b98 JavaThread "Timer-66" daemon [_thread_blocked, id=2569] 0xb0e76d10 JavaThread "Timer-65" daemon [_thread_blocked, id=2373] =>0xb1779350 JavaThread "Timer-64" daemon [_thread_in_native, id=2327] 0xaf438fd8 JavaThread "Timer-63" daemon [_thread_blocked, id=2311] 0xaa1f9808 JavaThread "Timer-62" daemon [_thread_blocked, id=1689] 0xaa11a980 JavaThread "Timer-61" daemon [_thread_blocked, id=1580] 0xafc71ef0 JavaThread "Timer-60" daemon [_thread_blocked, id=1377] 0xac4473c0 JavaThread "Timer-59" daemon [_thread_blocked, id=1166] 0xaadac790 JavaThread "Timer-58" daemon [_thread_blocked, id=757] 0xadd31318 JavaThread "Timer-57" daemon [_thread_blocked, id=560] 0xadf34f68 JavaThread "Timer-56" daemon [_thread_blocked, id=544] 0xaaf73608 JavaThread "Timer-55" daemon [_thread_blocked, id=483] 0xaad763a8 JavaThread "Timer-54" daemon [_thread_blocked, id=328] 0xaac2b048 JavaThread "Timer-53" daemon [_thread_blocked, id=32765] 0xac82ed18 JavaThread "Timer-52" daemon [_thread_blocked, id=32397] 0xa9b5ba48 JavaThread "Timer-51" daemon [_thread_blocked, id=32396] 0xaeae1f58 JavaThread "TP-Processor12" daemon [_thread_blocked, id=32395] 0xaeae1260 JavaThread "TP-Processor11" daemon [_thread_in_native, id=32394] 0xab1229e0 JavaThread "TP-Processor10" daemon [_thread_blocked, id=32393] 0xaebc3a80 JavaThread "TP-Processor9" daemon [_thread_blocked, id=32392] 0xac247498 JavaThread "Timer-50" daemon [_thread_blocked, id=32391] 0xb20692b8 JavaThread "Timer-49" daemon [_thread_blocked, id=32294] 0xb1e85460 JavaThread "Timer-48" daemon [_thread_blocked, id=31769] 0xac447c38 JavaThread "Timer-47" daemon [_thread_blocked, id=31605] 0xb1e736a8 JavaThread "Timer-46" daemon [_thread_blocked, id=31559] 0xaeb94da8 JavaThread "Timer-45" daemon [_thread_blocked, id=31468] 0xadd29c68 JavaThread "Timer-44" daemon [_thread_blocked, id=31362] 0xad9f0e30 JavaThread "Timer-43" daemon [_thread_blocked, id=31286] 0xae477790 JavaThread "Timer-42" daemon [_thread_blocked, id=31188] 0xaddffae8 JavaThread "Timer-41" daemon [_thread_blocked, id=30992] 0xb1ed2c18 JavaThread "Timer-40" daemon [_thread_blocked, id=30874] 0xafcd3048 JavaThread "Timer-39" daemon [_thread_blocked, id=30858] 0xb01c3f78 JavaThread "Timer-38" daemon [_thread_blocked, id=30767] 0xb215f738 JavaThread "Timer-37" daemon [_thread_blocked, id=30718] 0xb17507a8 JavaThread "Timer-36" daemon [_thread_blocked, id=30672] 0xaed2ba70 JavaThread "Timer-35" daemon [_thread_blocked, id=30598] 0xadf4fc80 JavaThread "Timer-34" daemon [_thread_blocked, id=30522] 0xaf426a58 JavaThread "Timer-33" daemon [_thread_blocked, id=30506] 0xab329008 JavaThread "Timer-32" daemon [_thread_blocked, id=30490] 0xaf48c558 JavaThread "Timer-31" daemon [_thread_blocked, id=30474] 0xb1ed9f00 JavaThread "Timer-30" daemon [_thread_blocked, id=30473] 0xaf5bc950 JavaThread "Timer-29" daemon [_thread_blocked, id=30427] 0xad8a0c28 JavaThread "Timer-28" daemon [_thread_blocked, id=30411] 0xad8b52f8 JavaThread "Timer-27" daemon [_thread_blocked, id=30380] 0xb0cab130 JavaThread "Timer-26" daemon [_thread_blocked, id=30251] 0xad88a908 JavaThread "Timer-25" daemon [_thread_blocked, id=30213] 0xb1e630a8 JavaThread "Timer-24" daemon [_thread_blocked, id=30212] 0xb0142448 JavaThread "Timer-23" daemon [_thread_blocked, id=30136] 0xad969f08 JavaThread "Timer-22" daemon [_thread_blocked, id=30120] 0xb012da98 JavaThread "Timer-21" daemon [_thread_blocked, id=30098] 0xaf603168 JavaThread "Timer-20" daemon [_thread_blocked, id=30088] 0xb0dae810 JavaThread "Timer-19" daemon [_thread_blocked, id=30071] 0xb01f1078 JavaThread "Timer-18" daemon [_thread_blocked, id=30070] 0xb0c6a850 JavaThread "Timer-17" daemon [_thread_blocked, id=30069] 0xb0c47ff0 JavaThread "Timer-16" daemon [_thread_blocked, id=30031] 0xb4068178 JavaThread "Timer-15" daemon [_thread_blocked, id=30030] 0xaf45cd70 JavaThread "Timer-14" daemon [_thread_blocked, id=30014] 0xaf8954a8 JavaThread "Timer-13" daemon [_thread_blocked, id=30013] 0xb0e45780 JavaThread "Timer-12" daemon [_thread_blocked, id=29997] 0xb4094308 JavaThread "Timer-11" daemon [_thread_blocked, id=29981] 0xb0161450 JavaThread "Timer-10" daemon [_thread_blocked, id=29965] 0xaf85de20 JavaThread "Timer-9" daemon [_thread_blocked, id=29949] 0xb041f908 JavaThread "Timer-8" daemon [_thread_blocked, id=29918] 0xb0ef7a60 JavaThread "Timer-7" daemon [_thread_blocked, id=29902] 0xb450e060 JavaThread "Timer-6" daemon [_thread_blocked, id=29865] 0xb018f980 JavaThread "Timer-5" daemon [_thread_blocked, id=29864] 0xb01a23b0 JavaThread "Timer-4" daemon [_thread_blocked, id=29863] 0xb2f12948 JavaThread "Timer-3" daemon [_thread_blocked, id=29847] 0xb41f86a0 JavaThread "Timer-2" daemon [_thread_in_native, id=29846] 0xb21c6738 JavaThread "Timer-1" daemon [_thread_blocked, id=29845] 0xb0e2a148 JavaThread "Timer-0" daemon [_thread_blocked, id=29844] 0xb209f0a8 JavaThread "TP-Processor8" daemon [_thread_blocked, id=29843] 0xb20ee7e8 JavaThread "TP-Processor7" daemon [_thread_blocked, id=29842] 0xb20edb90 JavaThread "TP-Processor6" daemon [_thread_blocked, id=29841] 0xb20e9a00 JavaThread "TP-Processor5" daemon [_thread_blocked, id=29840] 0xb1e36df8 JavaThread "TP-Monitor" daemon [_thread_blocked, id=29839] 0xb2f4adc8 JavaThread "TP-Processor4" daemon [_thread_in_native, id=29838] 0xb2fb4ba8 JavaThread "TP-Processor3" daemon [_thread_blocked, id=29837] 0xb1e3e3d0 JavaThread "TP-Processor2" daemon [_thread_blocked, id=29836] 0xb2b8e8c0 JavaThread "TP-Processor1" daemon [_thread_blocked, id=29835] 0xb21be7f8 JavaThread "ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon [_thread_blocked, id=29834] 0xb4101a30 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=29786] 0xb4100690 JavaThread "CompilerThread0" daemon [_thread_blocked, id=29785] 0xb44c0ce0 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=29784] 0xb44b7108 JavaThread "Finalizer" daemon [_thread_blocked, id=29783] 0xb44b5348 JavaThread "Reference Handler" daemon [_thread_blocked, id=29782] 0x08d7be28 JavaThread "main" [_thread_in_native, id=29779] Other Threads: 0xb44b2718 VMThread [id=29781] 0xb4102f50 WatcherThread [id=29787] VM state:not at safepoint (normal execution) VM Mutex/Monitor currently owned by a thread: None Heap def new generation total 72576K, used 53391K [0x08da0000, 0x0dc60000, 0x0dc60000) eden space 64512K, 82% used [0x08da0000, 0x0c1c3ee0, 0x0cca0000) from space 8064K, 0% used [0x0d480000, 0x0d480000, 0x0dc60000) to space 8064K, 0% used [0x0cca0000, 0x0cca0000, 0x0d480000) tenured generation total 967936K, used 753918K [0x0dc60000, 0x48da0000, 0x48da0000) the space 967936K, 77% used [0x0dc60000, 0x3bc9f8e0, 0x3bc9fa00, 0x48da0000) compacting perm gen total 476416K, used 476171K [0x48da0000, 0x65ee0000, 0x78da0000) the space 476416K, 99% used [0x48da0000, 0x65ea2cc8, 0x65ea2e00, 0x65ee0000) No shared spaces configured. Dynamic libraries: Can not get library information for pid = 2327 VM Arguments: jvm_args: -Dcatalina.home=/usr/local/tomcat -Djava.io.tmpdir=/var/tmp -Djava.awt.headless=true -Xmx1024m -XX:MaxPermSize=768m -DLD_ASSUME_KERNEL=2.4.1 java_command: <unknown> Environment Variables: PATH=/usr/kerberos/sbin:/usr/kerberos/bin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/usr/bin/X11:/usr/X11R6/bin:/root/bin USERNAME=root LD_LIBRARY_PATH=/usr/local/java/jre/lib/i386/client:/usr/local/java/jre/lib/i386 SHELL=/bin/bash Signal Handlers: SIGSEGV: [libjvm.so+0x2e52f0], sa_mask[0]=0x7ffbfeff, sa_flags=0x14000004 SIGBUS: [libjvm.so+0x2e52f0], sa_mask[0]=0x7ffbfeff, sa_flags=0x14000004 SIGFPE: [libjvm.so+0x25d640], sa_mask[0]=0x7ffbfeff, sa_flags=0x14000004 SIGPIPE: [libjvm.so+0x25d640], sa_mask[0]=0x7ffbfeff, sa_flags=0x14000004 SIGILL: [libjvm.so+0x25d640], sa_mask[0]=0x7ffbfeff, sa_flags=0x14000004 SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000 SIGUSR2: [libjvm.so+0x25f980], sa_mask[0]=0x00000000, sa_flags=0x14000004 SIGHUP: [jsvc.exec+0xe10], sa_mask[0]=0x00000001, sa_flags=0x14000000 SIGINT: [jsvc.exec+0xe10], sa_mask[0]=0x00000002, sa_flags=0x14000000 SIGQUIT: [libjvm.so+0x25f3b0], sa_mask[0]=0x7ffbfeff, sa_flags=0x14000004 SIGTERM: [jsvc.exec+0xe10], sa_mask[0]=0x00004000, sa_flags=0x14000000 --------------- S Y S T E M --------------- OS:Red Hat Enterprise Linux ES release 3 (Taroon Update 5) uname:Linux 2.4.21-27.0.2.ELsmp #1 SMP Wed Jan 12 23:35:44 EST 2005 i686 libc:glibc 2.3.2 NPTL 0.60 rlimit: STACK 10240k, CORE 0k, NPROC 7168, NOFILE 1024, AS infinity load average:0.45 0.16 0.10 CPU:total 4 family 15, cmov, cx8, fxsr, mmx, sse, sse2, ht Memory: 4k page, physical 2055392k(21640k free), swap 2096440k(1553976k free) vm_info: Java HotSpot(TM) Client VM (1.5.0_04-b05) for linux-x86, built on Jun 3 2005 03:17:33 by java_re with gcc 3.2.1-7a (J2SE release) REPRODUCIBILITY : It happens at random. ---------- BEGIN SOURCE ---------- Don't know as it happens in all different parts of the web application at random not always our application code sometimes in the jdk code, its completely random. ---------- END SOURCE ----------
If the VM is crashing it can't be a tomcat problem. I suggest you file a bug with Sun.
I already submitted this bug to Sun and they told me to send it to Apache for further Investigation ---- Message from Sun ---- Thank you for submitting a bug using our bug submit page. Please ask Apache to investigate this issue further. Since the Tomcat server appears to trigger this failure, the developers at Apache will be able to examine their product, isolate the cause of the behavior, and, if necessary, provide us with a detailed report. We greatly appreciate your efforts in identifying areas in the J2SE where we can improve upon and I would request you to continue doing so. Regards, Nelson
Lol about the broken OS being used, with NPTL, and sepcial mention for the VM argument "-DLD_ASSUME_KERNEL=2.4.1" :D
(In reply to comment #2) > I already submitted this bug to Sun and they told me to send it to Apache for > further Investigation We are now officially a garbage tray for Sun's support, cool :)
> We are now officially a garbage tray for Sun's support, cool :) I agree that this is most likely a Sun JDK bug but I am unable to get them to do anything about it. What I wasn't sure about is if using the LD_ASSUME_KERNEL could play any role in the problem as I have tried it with 2.4.1 and 2.2.5 as per different documents I found when google'n the problem. This is happening in our production enviroment so we need to find the solution from somebody. We are also looking at moving to JRockit JDK but jsvc crashes with that JDK. We seems to be quite SOL
In case you did not discern this from Remy, you need to set LD_ASSUME_KERNEL in the shell, not on the java command line. NPTL and the Sun vm have historically had trouble, hence the choice of a different threading library for stability.
I will give that a try and see if that helps. Thanks