[MAPREDUCE-139] JobTracker crashes Sun JVM

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Cannot Reproduce

Environment

OS:Red Hat Enterprise Linux AS release 4 (Nahant Update 2)

uname:Linux 2.6.9-22.12.y1-64 #5 SMP Fri Nov 4 13:39:56 PST 2005 x86_64
libc:glibc 2.3.4 NPTL 2.3.4
rlimit: STACK 10240k, CORE 0k, NPROC 32768, NOFILE 32768, AS infinity
load average:0.20 0.21 0.18

CPU:total 4 family 47, cmov, cx8, fxsr, mmx, sse, sse2, ht

Memory: 4k page, physical 988k(220k free), swap 3070k(3070k free)

vm_info: Java HotSpot(TM) Server VM (1.5.0_06-b05) for linux-x86, built on Nov 10 2005 10:56:33 by java_re with gcc 3.2.1-7a (J2SE release)

• Assignee: Owen O'Malley
• Reporter: Marco Nicosia
• Created: 28/Jul/06 23:03
• Updated: 16/Jan/12 09:43
• Resolved: 16/Jan/12 09:43

Description

From our jobtacker's .out file:

#

1. An unexpected error has been detected by HotSpot Virtual Machine:
   #
2. SIGSEGV (0xb) at pc=0xf27bf825, pid=24028, tid=1856498608
   #
3. Java VM: Java HotSpot(TM) Server VM (1.5.0_06-b05 mixed mode)
4. Problematic frame:
5. J org.mortbay.util.ByteArrayPool.getByteArray(I)[B
   #
6. An error report file with more information is saved as hs_err_pid24028.log
   #
7. If you would like to submit a bug report, please visit:
8. http://java.sun.com/webapps/bugreport/crash.jsp
   #

And the relevant parts of that hs_err_pid24028.log file:

#

1. An unexpected error has been detected by HotSpot Virtual Machine:
   #
2. SIGSEGV (0xb) at pc=0xf27bf825, pid=24028, tid=1856498608
   #
3. Java VM: Java HotSpot(TM) Server VM (1.5.0_06-b05 mixed mode)
4. Problematic frame:
5. J org.mortbay.util.ByteArrayPool.getByteArray(I)[B
   #

--------------- T H R E A D ---------------

Current thread (0x086e0088): JavaThread "SocketListener0-0" [_thread_in_Java, id=24112]

siginfo:si_signo=11, si_errno=0, si_code=2, si_addr=0x00022008

Registers:
EAX=0x00022000, EBX=0x75a0fac8, ECX=0x00000000, EDX=0x75b31258
ESP=0x6ea7ded0, EBP=0x6ea7df3c, ESI=0x71b6def5, EDI=0x00000402
EIP=0xf27bf825, CR2=0x00022008, EFLAGS=0x00010217

Top of Stack: (sp=0x6ea7ded0)
0x6ea7ded0: 71549561 6ea7def0 00002000 00000400
0x6ea7dee0: 339dc50d 086e0088 6ea7df3c 6ea7df50
0x6ea7def0: 00000000 71b6def5 7154d7c9 f2595a80
0x6ea7df00: 6ea7df3c 6ea7df50 7154d7d8 6ea7df14
0x6ea7df10: 6ea7df3c f2501a14 00002000 e57bd350
0x6ea7df20: 6ea7df20 71b6def5 6ea7df50 71b71798
0x6ea7df30: 00000000 71b6df80 6ea7df44 6ea7df78
0x6ea7df40: f2501aeb 719431e8 00002000 e57bd2d0

Instructions: (pc=0xf27bf825)
0xf27bf815: 8b 98 a4 00 00 00 8b 4b 20 89 4c 24 20 8b 41 08
0xf27bf825: 8b 70 08 8b de 4b 8b 4c 24 10 23 cb 89 4c 24 10

Stack: [0x6e9fe000,0x6ea7f000), sp=0x6ea7ded0, free space=511k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
J org.mortbay.util.ByteArrayPool.getByteArray(I)[B
j org.mortbay.http.HttpInputStream.<init>(Ljava/io/InputStream;I)V+15
j org.mortbay.http.HttpConnection.<init>(Lorg/mortbay/http/HttpListener;Ljava/net/InetAddress;Ljava/io/InputStream;Ljava/io/OutputStream;Ljava/lang/Object;)V+97
j org.mortbay.http.SocketListener.createConnection(Ljava/net/Socket;)Lorg/mortbay/http/HttpConnection;+18
j org.mortbay.http.SocketListener.handleConnection(Ljava/net/Socket;)V+2
j org.mortbay.util.ThreadedServer.handle(Ljava/lang/Object;)V+19

--------------- P R O C E S S ---------------

Java Threads: ( => current thread )
0x086bb8a0 JavaThread "Thread-0" [_thread_blocked, id=24117]
0x0859dd38 JavaThread "Thread-20" [_thread_blocked, id=24116]
0x0859a8a8 JavaThread "Thread-19" [_thread_blocked, id=24115]
0x0859a1f8 JavaThread "Thread-18" [_thread_blocked, id=24114]
0x085d54d8 JavaThread "SocketListener0-1" [_thread_in_native, id=24113]
=>0x086e0088 JavaThread "SocketListener0-0" [_thread_in_Java, id=24112]

[...]

Heap
PSYoungGen total 21248K, used 19699K [0xe4640000, 0xe6a00000, 0xf2470000)
eden space 19584K, 92% used [0xe4640000,0xe57def08,0xe5960000)
from space 1664K, 99% used [0xe5960000,0xe5afe020,0xe5b00000)
to space 8384K, 0% used [0xe61d0000,0xe61d0000,0xe6a00000)
PSOldGen total 390080K, used 370693K [0x75470000, 0x8d160000, 0xe4640000)
object space 390080K, 95% used [0x75470000,0x8be716e0,0x8d160000)
PSPermGen total 16384K, used 8080K [0x71470000, 0x72470000, 0x75470000)
object space 16384K, 49% used [0x71470000,0x71c54298,0x72470000)

[...]

VM Arguments:
jvm_args: -Xmx2000m -Dhadoop.log.dir=/export//current/../run2/log -Dhadoop.log.file=hadoop-jobtracker.log -Dhadoop.home.dir=/export/current -Dhadoop.id.str=crawler -Dhadoop.root.logger=INFO,DRFA
java_command: org.apache.hadoop.mapred.JobTracker
Launcher Type: SUN_STANDARD

Environment Variables:
JAVA_HOME=/export//java/jdk
PATH=/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin:/export//current/bin
LD_LIBRARY_PATH=/export/java/jdk1.5.0_06/jre/lib/i386/server:/export//java/jdk1.5.0_06/jre/lib/i386:/export//java/jdk1.5.0_06/jre/../lib/i386
SHELL=/bin/tcsh
HOSTTYPE=x86_64-linux
OSTYPE=linux
MACHTYPE=x86_64