Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
1.4.0
-
None
-
None
Description
Introduction of the new hook-based accounting for the process memory brought by b15ea202d6a2be427614a876aa56d03b9c8b3a1b causes issues on OS X. This manifests itself in many failed tests. Usually, a process ends up killed by SIGILL. If running under debugger, it looks like the following:
* thread #81: tid = 0x10470b6, 0x00000001039897dc libtcmalloc.4.dylib`(anonymous namespace)::do_malloc_no_errno(unsigned long) [inlined] tcmalloc::ThreadCache::GetThreadHeap() + 10 at thread_cache.h:403, stop reason = EXC_BAD_ACCESS (code=2, address=0x109952c88)
frame #0: 0x00000001039897dc libtcmalloc.4.dylib`(anonymous namespace)::do_malloc_no_errno(unsigned long) [inlined] tcmalloc::ThreadCache::GetThreadHeap() + 10 at thread_cache.h:403
400 return threadlocal_data_.heap;
401 #endif
402 return reinterpret_cast<ThreadCache *>(
-> 403 perftools_pthread_getspecific(heap_key_));
404 }
405
406 inline ThreadCache* ThreadCache::GetCacheWhichMustBePresent() {
with very deep call stack:
frame #2462: 0x00007fff9456f187 libsystem_malloc.dylib`malloc + 42
frame #2463: 0x00007fff891f0da0 libdyld.dylib`tlv_allocate_and_initialize_for_key + 323
frame #2464: 0x00007fff891f1549 libdyld.dylib`tlv_get_addr + 296
frame #2465: 0x0000000103f276e1 libkudu_util.dylib`kudu::Striped64::get_tls_hashcode() + 17 at striped64.cc:55
frame #2466: 0x0000000103f27d75 libkudu_util.dylib`kudu::LongAdder::IncrementBy(this=0x000000010711cfc0, x=80) + 85 at striped64.cc:146
frame #2467: 0x0000000103f14b7f libkudu_util.dylib`kudu::process_memory::(anonymous namespace)::NewHook(ptr=0x000000010bf08af0, size=80) + 79 at process_memory.cc:187
frame #2468: 0x000000010420a8e8 libtcmalloc.4.dylib`MallocHook::InvokeNewHookSlow(p=0x000000010bf08af0, s=80) + 184 at malloc_hook.cc:514
frame #2469: 0x0000000104218f4f libtcmalloc.4.dylib`tc_malloc + 239 at libc_override_osx.h:211
frame #2470: 0x00007fff94570667 libsystem_malloc.dylib`malloc_zone_malloc + 71
frame #2471: 0x00007fff9456f187 libsystem_malloc.dylib`malloc + 42
frame #2472: 0x00007fff891f0da0 libdyld.dylib`tlv_allocate_and_initialize_for_key + 323
frame #2473: 0x00007fff891f1549 libdyld.dylib`tlv_get_addr + 296
frame #2474: 0x0000000103f276e1 libkudu_util.dylib`kudu::Striped64::get_tls_hashcode() + 17 at striped64.cc:55
frame #2475: 0x0000000103f27d75 libkudu_util.dylib`kudu::LongAdder::IncrementBy(this=0x000000010711cfc0, x=80) + 85 at striped64.cc:146
frame #2476: 0x0000000103f14b7f libkudu_util.dylib`kudu::process_memory::(anonymous namespace)::NewHook(ptr=0x000000010bf08aa0, size=80) + 79 at process_memory.cc:187
frame #2477: 0x000000010420a8e8 libtcmalloc.4.dylib`MallocHook::InvokeNewHookSlow(p=0x000000010bf08aa0, s=80) + 184 at malloc_hook.cc:514
frame #2478: 0x0000000104218f4f libtcmalloc.4.dylib`tc_malloc + 239 at libc_override_osx.h:211
frame #2479: 0x00007fff94570667 libsystem_malloc.dylib`malloc_zone_malloc + 71
frame #2480: 0x00007fff9456f187 libsystem_malloc.dylib`malloc + 42
frame #2481: 0x00007fff891f0da0 libdyld.dylib`tlv_allocate_and_initialize_for_key + 323
frame #2482: 0x00007fff891f1549 libdyld.dylib`tlv_get_addr + 296
frame #2483: 0x0000000103f276e1 libkudu_util.dylib`kudu::Striped64::get_tls_hashcode() + 17 at striped64.cc:55
frame #2484: 0x0000000103f27d75 libkudu_util.dylib`kudu::LongAdder::IncrementBy(this=0x000000010711cfc0, x=80) + 85 at striped64.cc:146
frame #2485: 0x0000000103f14b7f libkudu_util.dylib`kudu::process_memory::(anonymous namespace)::NewHook(ptr=0x000000010bf08a50, size=80) + 79 at process_memory.cc:187
frame #2486: 0x000000010420a8e8 libtcmalloc.4.dylib`MallocHook::InvokeNewHookSlow(p=0x000000010bf08a50, s=80) + 184 at malloc_hook.cc:514
frame #2487: 0x0000000104218f4f libtcmalloc.4.dylib`tc_malloc + 239 at libc_override_osx.h:211
frame #2488: 0x00007fff94570667 libsystem_malloc.dylib`malloc_zone_malloc + 71
frame #2489: 0x00007fff9456f187 libsystem_malloc.dylib`malloc + 42
frame #2490: 0x00007fff891f0da0 libdyld.dylib`tlv_allocate_and_initialize_for_key + 323
frame #2491: 0x00007fff891f1549 libdyld.dylib`tlv_get_addr + 296
frame #2492: 0x0000000103f276e1 libkudu_util.dylib`kudu::Striped64::get_tls_hashcode() + 17 at striped64.cc:55
frame #2493: 0x0000000103f27d75 libkudu_util.dylib`kudu::LongAdder::IncrementBy(this=0x000000010711cfc0, x=80) + 85 at striped64.cc:146
frame #2494: 0x0000000103f14b7f libkudu_util.dylib`kudu::process_memory::(anonymous namespace)::NewHook(ptr=0x000000010bf08a00, size=80) + 79 at process_memory.cc:187
frame #2495: 0x000000010420a8e8 libtcmalloc.4.dylib`MallocHook::InvokeNewHookSlow(p=0x000000010bf08a00, s=80) + 184 at malloc_hook.cc:514
frame #2496: 0x0000000104218f4f libtcmalloc.4.dylib`tc_malloc + 239 at libc_override_osx.h:211
frame #2497: 0x00007fff94570667 libsystem_malloc.dylib`malloc_zone_malloc + 71
frame #2498: 0x00007fff9456f187 libsystem_malloc.dylib`malloc + 42
frame #2499: 0x00007fff891f0da0 libdyld.dylib`tlv_allocate_and_initialize_for_key + 323
frame #2500: 0x00007fff891f1549 libdyld.dylib`tlv_get_addr + 296
frame #2501: 0x0000000103f276e1 libkudu_util.dylib`kudu::Striped64::get_tls_hashcode() + 17 at striped64.cc:55
frame #2502: 0x0000000103f27d75 libkudu_util.dylib`kudu::LongAdder::IncrementBy(this=0x000000010711cfc0, x=80) + 85 at striped64.cc:146
frame #2503: 0x0000000103f14b7f libkudu_util.dylib`kudu::process_memory::(anonymous namespace)::NewHook(ptr=0x000000010bf089b0, size=80) + 79 at process_memory.cc:187
frame #2504: 0x000000010420a8e8 libtcmalloc.4.dylib`MallocHook::InvokeNewHookSlow(p=0x000000010bf089b0, s=80) + 184 at malloc_hook.cc:514
frame #2505: 0x0000000104218f4f libtcmalloc.4.dylib`tc_malloc + 239 at libc_override_osx.h:211
frame #2506: 0x00007fff94570667 libsystem_malloc.dylib`malloc_zone_malloc + 71
frame #2507: 0x00007fff9456f187 libsystem_malloc.dylib`malloc + 42
This is specific only to OS X builds, Linux builds do not have such an issue.
Since we want to switch to jemalloc in the long run, we disabled tcmalloc for OS X builds as a quick fix.
Some related info:
https://github.com/gperftools/gperftools/issues/416