[IMPALA-5414] Impalad aborting in tcmalloc during JVM thread start

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: Impala 2.9.0
• Component/s: Backend, Frontend
• Labels: bugbash-2017-05-31
• Epic Color: ghx-label-4
• Assignee: Unassigned
• Reporter: Lars Volker
• Created: 01/Jun/17 00:54
• Updated: 05/Jun/17 17:37

Description

This was found during today's bug bash. Logfiles of both affected impalads contain many lines like the following before writing a minidump. Logfiles and resolved minidumps are attached.

I0531 17:01:20.666857 31217 thrift-util.cc:111] TAcceptQueueServer client died: SSL_read: Connection reset by peer
I0531 17:01:20.670058 31232 thrift-util.cc:111] TAcceptQueueServer client died: SSL_read: Connection reset by peer
I0531 17:01:20.673490 31238 thrift-util.cc:111] TAcceptQueueServer client died: SSL_read: Connection reset by peer
I0531 17:01:20.676220 31172 thrift-util.cc:111] TAcceptQueueServer client died: SSL_read: Connection reset by peer
I0531 17:01:20.679427 31245 thrift-util.cc:111] TAcceptQueueServer client died: SSL_read: Connection reset by peer
I0531 17:01:20.682533 31253 thrift-util.cc:111] TAcceptQueueServer client died: SSL_read: Connection reset by peer

The crashing thread looks like this:

Crash reason:  SIGABRT
Crash address: 0x1e000002614
Process uptime: not available

Thread 64 (crashed)
 0  libc-2.12.so + 0x32625
    rax = 0x0000000000000000   rdx = 0x0000000000000006
    rcx = 0xffffffffffffffff   rbx = 0x00007f48b2f66000
    rsi = 0x00000000000026ab   rdi = 0x0000000000002614
    rbp = 0x0000000001cd7000   rsp = 0x00007f48b9402478
     r8 = 0x0000000000004ba8    r9 = 0xff092d63646b6860
    r10 = 0x0000000000000008   r11 = 0x0000000000000206
    r12 = 0x0000000001c693c7   r13 = 0x0000000002483c40
    r14 = 0x00000000ffffffff   r15 = 0x00000000037eefe8
    rip = 0x00000038d3432625
    Found by: given as instruction pointer in context
 1  libc-2.12.so + 0x33e05
    rsp = 0x00007f48b9402480   rip = 0x00000038d3433e05
    Found by: stack scanning
 2  impalad!_fini + 0x512d7
    rsp = 0x00007f48b9402488   rip = 0x0000000001c693c7
    Found by: stack scanning
 3  libc-2.12.so + 0x1572bc
    rsp = 0x00007f48b9402490   rip = 0x00000038d35572bc
    Found by: stack scanning
 4  libc-2.12.so + 0x66216
    rsp = 0x00007f48b94024b0   rip = 0x00000038d3466216
    Found by: stack scanning
 5  libc-2.12.so + 0x4f348
    rsp = 0x00007f48b94024d0   rip = 0x00000038d344f348
    Found by: stack scanning
 6  libc-2.12.so + 0x155bac
    rsp = 0x00007f48b9402510   rip = 0x00000038d3555bac
    Found by: stack scanning
 7  impalad!_fini + 0xbef10
    rsp = 0x00007f48b9402518   rip = 0x0000000001cd7000
    Found by: stack scanning
 8  libc-2.12.so + 0xdb589
    rsp = 0x00007f48b9402520   rip = 0x00000038d34db589
    Found by: stack scanning
 9  impalad!_fini + 0xbef10
    rsp = 0x00007f48b94025a8   rip = 0x0000000001cd7000
    Found by: stack scanning
10  libc-2.12.so + 0x2b74e
    rsp = 0x00007f48b94025b0   rip = 0x00000038d342b74e
    Found by: stack scanning
11  impalad!_fini + 0x86bb50
    rsp = 0x00007f48b94025b8   rip = 0x0000000002483c40
    Found by: stack scanning
12  libc-2.12.so + 0x155bac
    rsp = 0x00007f48b94025c0   rip = 0x00000038d3555bac
    Found by: stack scanning
13  impalad!_fini + 0x512d7
    rsp = 0x00007f48b94025c8   rip = 0x0000000001c693c7
    Found by: stack scanning
14  impalad!_fini + 0x86bb50
    rsp = 0x00007f48b94025d8   rip = 0x0000000002483c40
    Found by: stack scanning
15  impalad!_fini + 0x512d7
    rsp = 0x00007f48b94025f8   rip = 0x0000000001c693c7
    Found by: stack scanning
16  impalad!_fini + 0xbef10
    rsp = 0x00007f48b9402600   rip = 0x0000000001cd7000
    Found by: stack scanning
17  libc-2.12.so + 0x2b810
    rsp = 0x00007f48b9402610   rip = 0x00000038d342b810
    Found by: stack scanning
18  impalad!_fini + 0x512d7
    rsp = 0x00007f48b9402618   rip = 0x0000000001c693c7
    Found by: stack scanning
19  impalad!boost::shared_array<apache::thrift::concurrency::Mutex>::~shared_array() + 0x70
    rsp = 0x00007f48b9402630   rip = 0x0000000001b3f330
    Found by: stack scanning
20  impalad!boost::detail::sp_counted_impl_p<apache::thrift::concurrency::Mutex::impl>::dispose() + 0x4f
    rsp = 0x00007f48b9402640   rip = 0x0000000001b4f23f
    Found by: stack scanning
21  impalad!boost::detail::sp_counted_impl_pd<apache::thrift::concurrency::Mutex*, boost::checked_array_deleter<apache::thrift::concurrency::Mutex> >::dispose() + 0xaa
    rbx = 0x000000000368c910   rsp = 0x00007f48b9402650
    rip = 0x0000000001b3f4ea
    Found by: call frame info
22  impalad!boost::shared_array<apache::thrift::concurrency::Mutex>::~shared_array() + 0x39
    rbx = 0x000000000373a940   rbp = 0x00000000ffffffff
    rsp = 0x00007f48b9402680   r12 = 0x000000000372ae40
    r13 = 0x00000000037eec00   r14 = 0x00000000000003d8
    rip = 0x0000000001b3f2f9
    Found by: call frame info
23  libc-2.12.so + 0x35b22
    rbx = 0x00000000036df180   rbp = 0x000000000000008f
    rsp = 0x00007f48b94026a0   r12 = 0x000000000372ae40
    r13 = 0x00000000037eec00   r14 = 0x00000000000003d8
    rip = 0x00000038d3435b22
    Found by: call frame info
24  libjvm.so!vm_direct_exit(int) + 0x17
    rsp = 0x00007f48b94026c0   rip = 0x00007f490b80bce7
    Found by: stack scanning
25  libjvm.so!VM_Operation::evaluate() + 0x55
    rsp = 0x00007f48b94026e0   rip = 0x00007f490bbb68b5
    Found by: stack scanning
26  libjvm.so!VMThread::evaluate_operation(VM_Operation*) + 0xba
    rsp = 0x00007f48b9402730   rip = 0x00007f490bbb4c7a
    Found by: stack scanning
27  libjvm.so!VMThread::loop() + 0x1ce
    rsp = 0x00007f48b94027e0   rip = 0x00007f490bbb4ffe
    Found by: stack scanning
28  impalad!tcmalloc::ThreadCache::FetchFromCentralCache(unsigned long, unsigned long) + 0x63
    rsp = 0x00007f48b9402840   rip = 0x0000000001c07303
    Found by: stack scanning
29  impalad!malloc + 0x27b
    rsp = 0x00007f48b94028a0   rip = 0x0000000001c142ab
    Found by: stack scanning
30  libjvm.so!os::malloc(unsigned long, unsigned short, unsigned char*) + 0x6f
    rsp = 0x00007f48b94028e0   rip = 0x00007f490ba25a5f
    Found by: stack scanning
31  libjvm.so!os::current_stack_size() + 0x15
    rsp = 0x00007f48b94028f0   rip = 0x00007f490ba327f5
    Found by: stack scanning
32  libjvm.so!VMThread::run() + 0x70
    rsp = 0x00007f48b9402980   rip = 0x00007f490bbb5470
    Found by: stack scanning
33  libjvm.so!Monitor::wait(bool, long, bool) + 0x22e
    rsp = 0x00007f48b94029a0   rip = 0x00007f490b9f023e
    Found by: stack scanning
34  libjvm.so!java_start(Thread*) + 0x108
    rsp = 0x00007f48b94029f0   rip = 0x00007f490ba2e988
    Found by: stack scanning