Uploaded image for project: 'Kudu'
  1. Kudu
  2. KUDU-3405

Tabelt Server become 'Zombie Process'

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 1.13.0
    • None
    • tserver
    • None
    • linux-4.18.0-147.8.1.el7.aarch64
      gcc-4.8.5
      docker-19.03.4
      kudu-1.13.0(master*3，tserver*11)

    Description

      In ARM, running KUDU. Tablet server incidental 'Zombie Process':

      1. Process is exist;

      2.Port 7050/8050 exist;

      3.Log print is stopped;

      4.minidmp is printed;

      5.other tserver failed to connect it;

      6.master failed to connect it, and loss of heartbeat;

      7.memory/cpu/disk is normal;

       

      example-1(2022-11-17 add):

      root         7     1  4 02:44 ?        00:20:47 /opt/java/kudu/release/bin/kudu-tserver --flagfile=/opt/java/kudu/conf/tserver.gflagfile
root     25428     7  0 02:55 ?        00:00:01 [rpc worker-89] <defunct>

       

      example-2(2022-11-17 add):

       

      root         7     1 72 Nov16 ?        13:52:49 /opt/java/kudu/release/bin/kudu-tserver --flagfile=/opt/java/kudu/conf/tserver.gflagfile
root     40977     7  0 Nov16 ?        00:00:01 [rpc reactor-23] <defunct>

       

      example-3(2022-11-17 add):

       

      root         7     1  1 Nov16 ?        00:14:27 /opt/java/kudu/release/bin/kudu-tserver --flagfile=/opt/java/kudu/conf/tserver.gflagfile
root     13818     7  0 Nov16 ?        00:00:01 [rpc worker-162] <defunct>

       

       

      minidmp:

      ${KUDU_HOME}/thirdparty/installed/uninstrumented/bin/minidump-2-core -o xxx.core xxx.dmp

      gdb bin/kudu-tserver xxx.core

      (gdb) bt

       

      GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-120.el7
      Copyright (C) 2013 Free Software Foundation, Inc.
      License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
      and "show warranty" for details.
      This GDB was configured as "aarch64-redhat-linux-gnu".
      For bug reporting instructions, please see:
      <http://www.gnu.org/software/gdb/bugs/>...
      Reading symbols from /opt/kudu-1.13.0-aarch64/kudu-1.13.0-aarch64/release/bin/kudu-tserver...done.

      warning: core file may not match specified executable file.
      [New LWP 44946]
      [New LWP 7]
      [New LWP 9]
      [New LWP 10]
      [New LWP 11]
      [New LWP 12]
      [New LWP 13]
      [New LWP 14]
      [New LWP 85]
      [New LWP 86]
      [New LWP 87]
      [New LWP 88]
      [New LWP 89]
      [New LWP 90]
      [New LWP 91]
      [New LWP 92]
      [New LWP 93]
      [New LWP 148]
      [New LWP 149]
      [New LWP 150]
      [New LWP 151]
      [New LWP 152]
      [New LWP 153]
      [New LWP 154]
      [New LWP 155]
      [New LWP 156]
      [New LWP 157]
      [New LWP 158]
      [New LWP 159]
      [New LWP 160]
      [New LWP 161]
      [New LWP 162]
      [New LWP 163]
      [New LWP 164]
      [New LWP 165]
      [New LWP 166]
      [New LWP 167]
      [New LWP 168]
      [New LWP 169]
      [New LWP 170]
      [New LWP 171]
      [New LWP 172]
      [New LWP 173]
      [New LWP 174]
      [New LWP 175]
      [New LWP 176]
      [New LWP 177]
      [New LWP 178]
      [New LWP 179]
      [New LWP 180]
      [New LWP 181]
      [New LWP 182]
      [New LWP 183]
      [New LWP 184]
      [New LWP 185]
      [New LWP 186]
      [New LWP 187]
      [New LWP 188]
      [New LWP 189]
      [New LWP 190]
      [New LWP 191]
      [New LWP 192]
      [New LWP 193]
      [New LWP 194]
      [New LWP 195]
      [New LWP 196]
      [New LWP 197]
      [New LWP 198]
      [New LWP 199]
      [New LWP 200]
      [New LWP 201]
      [New LWP 202]
      [New LWP 203]
      [New LWP 204]
      [New LWP 205]
      [New LWP 206]
      [New LWP 207]
      [New LWP 208]
      [New LWP 209]
      [New LWP 210]
      [New LWP 211]
      [New LWP 212]
      [New LWP 213]
      [New LWP 214]
      [New LWP 215]
      [New LWP 216]
      [New LWP 217]
      [New LWP 218]
      [New LWP 219]
      [New LWP 220]
      [New LWP 221]
      [New LWP 222]
      [New LWP 223]
      [New LWP 224]
      [New LWP 225]
      [New LWP 226]
      [New LWP 227]
      [New LWP 228]
      [New LWP 229]
      [New LWP 230]
      [New LWP 231]
      [New LWP 232]
      [New LWP 233]
      [New LWP 234]
      [New LWP 235]
      [New LWP 236]
      [New LWP 237]
      [New LWP 238]
      [New LWP 239]
      [New LWP 240]
      [New LWP 241]
      [New LWP 242]
      [New LWP 243]
      [New LWP 244]
      [New LWP 245]
      [New LWP 246]
      [New LWP 247]
      [New LWP 248]
      [New LWP 249]
      [New LWP 250]
      [New LWP 251]
      [New LWP 252]
      [New LWP 253]
      [New LWP 254]
      [New LWP 255]
      [New LWP 884]
      [New LWP 44893]
      [New LWP 44901]
      [New LWP 44919]
      [New LWP 44922]
      [New LWP 44923]
      [New LWP 44924]
      [New LWP 44925]
      [New LWP 44926]
      [New LWP 44927]
      [New LWP 44928]
      [New LWP 44929]
      [New LWP 44930]
      [New LWP 44931]
      [New LWP 44932]
      [New LWP 44933]
      [New LWP 44934]
      [New LWP 44935]
      [New LWP 44936]
      [New LWP 44937]
      [New LWP 44938]
      [New LWP 44939]
      [New LWP 44940]
      [New LWP 44941]
      [New LWP 44942]
      [New LWP 44943]
      [New LWP 44944]
      [New LWP 44945]
      [New LWP 44947]

      warning: Could not load shared library symbols for linux-gate.so.
      Do you need "set solib-search-path" or "set sysroot"?
      Warning: couldn't activate thread debugging using libthread_db: Cannot find new threads: generic error

      warning: File "/usr/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load:/usr/bin/mono-gdb.py".
      To enable execution of this file add
          add-auto-load-safe-path /usr/lib64/libthread_db-1.0.so
      line to your configuration file "/root/.gdbinit".
      To completely disable this security protection add
          set auto-load safe-path /
      line to your configuration file "/root/.gdbinit".
      For more information about this security protection see the
      "Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
          info "(gdb)Auto-loading safe path"

      warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
      Warning: couldn't activate thread debugging using libthread_db: Cannot find new threads: generic error

      warning: File "/usr/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load:/usr/bin/mono-gdb.py".

      warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
      Failed to read a valid object file image from memory.
      Core was generated by `/kudu/release/bin/kudu-tserver --flagfile=/opt/kudu/conf/tserver.'.
      Program terminated with signal 11, Segmentation fault.
      #0  dwarf_get (c=0xffffa43dce20, val=<synthetic pointer>, loc=...) at ${KUDU_HOME}/thirdparty/src/libunwind-1.4.0/include/tdep-aarch64/libunwind_i.h:146
      146    ${KUDU_HOME}/thirdparty/src/libunwind-1.4.0/include/tdep-aarch64/libunwind_i.h: No such file or directory.
      Missing separate debuginfos, use: debuginfo-install cyrus-sasl-gs2-2.1.26-23.el7.aarch64 cyrus-sasl-gssapi-2.1.26-23.el7.aarch64 cyrus-sasl-ldap-2.1.26-23.el7.aarch64 cyrus-sasl-lib-2.1.26-23.el7.aarch64 cyrus-sasl-md5-2.1.26-23.el7.aarch64 cyrus-sasl-ntlm-2.1.26-23.el7.aarch64 cyrus-sasl-plain-2.1.26-23.el7.aarch64 cyrus-sasl-scram-2.1.26-23.el7.aarch64 cyrus-sasl-sql-2.1.26-23.el7.aarch64 glibc-2.17-317.el7.aarch64 keyutils-libs-1.5.8-3.el7.aarch64 krb5-libs-1.15.1-50.el7.aarch64 libcom_err-1.42.9-19.el7.aarch64 libdb-5.3.21-25.el7.aarch64 libgcc-4.8.5-44.el7.aarch64 libselinux-2.5-15.el7.aarch64 libstdc++-4.8.5-39.el7.aarch64 mariadb-libs-5.5.68-1.el7.aarch64 ncurses-libs-5.9-14.20130511.el7_4.aarch64 nspr-4.21.0-1.el7.aarch64 nss-3.44.0-7.el7_7.aarch64 nss-softokn-freebl-3.44.0-8.el7_7.aarch64 nss-util-3.44.0-4.el7_7.aarch64 openldap-2.4.44-21.el7_6.aarch64 openssl-libs-1.0.2k-19.el7.aarch64 pcre-8.32-17.el7.aarch64 postgresql-libs-9.2.24-4.el7_8.aarch64 zlib-1.2.7-18.el7.aarch64
      (gdb) bt
      #0  dwarf_get (c=0xffffa43dce20, val=<synthetic pointer>, loc=...) at ${KUDU_HOME}/thirdparty/src/libunwind-1.4.0/include/tdep-aarch64/libunwind_i.h:146
      #1  apply_reg_state (c=0xffffa43dce20, c@entry=<error reading variable: Cannot access memory at address 0xffffa43dbef8>, rs=<error reading variable: Cannot access memory at address 0xffffa43dbf68>)
          at ${KUDU_HOME}/thirdparty/src/libunwind-1.4.0/src/dwarf/Gparser.c:878
      (gdb) 

      gdb -p [pid]: (Extract section)

      gdb -p [pid]

      (gdb) info threads

      (gdb) thread apply all bt

      case 1:

      ... 

      148  Thread 0xffffa79def90 (LWP 314) "excess-log-dele" 0x0000ffffb2ecf008 in __lll_lock_wait () from /lib64/libpthread.so.0

       ...

        13   Thread 0xffff9e0aef90 (LWP 14437) "raft [worker]-1" 0x000000000208358c in sys_wait4 (o=1073741824, r=0x0, s=0xffff9e0ab6f4, p=<optimized out>)
          at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/breakpad-9eac2058b70615519b2c4d8c6bdbfca1bd079e39/src/third_party/lss/linux_syscall_support.h:3692

      ...

       

      ...

      Thread 13 (Thread 0xffff9e0aef90 (LWP 14437)):
      #0  0x000000000208358c in sys_wait4 (o=1073741824, r=0x0, s=0xffff9e0ab6f4, p=<optimized out>) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/breakpad-9eac2058b70615519b2c4d8c6bdbfca1bd079e39/src/third_party/lss/linux_syscall_support.h:3692
      #1  sys_waitpid (options=1073741824, status=0xffff9e0ab6f4, pid=<optimized out>) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/breakpad-9eac2058b70615519b2c4d8c6bdbfca1bd079e39/src/third_party/lss/linux_syscall_support.h:3697
      #2  google_breakpad::ExceptionHandler::GenerateDump (this=0x3515c4d0, context=<optimized out>) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/breakpad-9eac2058b70615519b2c4d8c6bdbfca1bd079e39/src/client/linux/handler/exception_handler.cc:553
      #3  0x00000000020838fc in google_breakpad::ExceptionHandler::SignalHandler (sig=11, info=0xffff9e0ab810, uc=0xffff9e0ab890)
          at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/breakpad-9eac2058b70615519b2c4d8c6bdbfca1bd079e39/src/client/linux/handler/exception_handler.cc:376
      #4  <signal handler called>
      #5  access_mem (as=0x2dc6220 <local_addr_space>, addr=1, val=0xffff9e0aca98, write=0, arg=0xffff9e0acb40) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/libunwind-1.4.0/src/aarch64/Ginit.c:96
      #6  0x0000000002066b6c in _ULaarch64_is_signal_frame (cursor=cursor@entry=0xffff9e0ace20) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/libunwind-1.4.0/src/aarch64/Gis_signal_frame.c:51
      #7  0x0000000002063d18 in _ULaarch64_step (cursor=0xffff9e0ace20) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/libunwind-1.4.0/src/aarch64/Gstep.c:143
      #8  0x0000000001f88fc0 in kudu::StackTrace::Collect (this=this@entry=0x54007dd8, skip_frames=0, skip_frames@entry=1) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/util/debug-util.cc:612
      #9  0x0000000001f8948c in Collect (skip_frames=1, this=0x54007dd8) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/util/debug-util.cc:288
      #10 kudu::(anonymous namespace)::HandleStackTraceSignal (info=<optimized out>) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/util/debug-util.cc:286
      #11 <signal handler called>
      #12 0x0000ffffb271f640 in clone () from /lib64/libc.so.6
      #13 0x0000ffffb2ec6d78 in do_clone.constprop.3 () from /lib64/libpthread.so.0
      #14 0x0000ffffb2ec84f8 in pthread_create@@GLIBC_2.17 () from /lib64/libpthread.so.0
      #15 0x0000000000000001 in ?? ()
      Backtrace stopped: previous frame identical to this frame (corrupt stack?)

       ...

      Thread 148 (Thread 0xffffa79def90 (LWP 314)):
      #0  0x0000ffffb2ecf008 in __lll_lock_wait () from /lib64/libpthread.so.0
      #1  0x0000ffffb2ec9c40 in pthread_mutex_lock () from /lib64/libpthread.so.0
      #2  0x0000000002065c14 in get_rs_cache (saved_maskp=0xffffa79dbf60, as=<optimized out>) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/libunwind-1.4.0/src/dwarf/Gparser.c:610
      #3  find_reg_state (sr=0xffffa79dbfe0, c=0xffffa79dca80) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/libunwind-1.4.0/src/dwarf/Gparser.c:908
      #4  _ULaarch64_dwarf_step (c=c@entry=0xffffa79dca80) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/libunwind-1.4.0/src/dwarf/Gparser.c:965
      #5  0x0000000002063ef4 in _ULaarch64_step (cursor=0xffffa79dca80) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/libunwind-1.4.0/src/aarch64/Gstep.c:146
      #6  0x0000000001f88fc0 in kudu::StackTrace::Collect (this=this@entry=0x54002838, skip_frames=0, skip_frames@entry=1) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/util/debug-util.cc:612
      #7  0x0000000001f8948c in Collect (skip_frames=1, this=0x54002838) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/util/debug-util.cc:288
      #8  kudu::(anonymous namespace)::HandleStackTraceSignal (info=<optimized out>) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/util/debug-util.cc:286
      #9  <signal handler called>
      #10 0x0000ffffb2ecc420 in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
      #11 0x0000000001f88bc8 in kudu::ConditionVariable::WaitUntil (this=this@entry=0xffffcf707510, until=...) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/util/condition_variable.cc:87
      #12 0x0000000000a80f54 in WaitUntil (when=..., this=0xffffcf7074e0) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/util/countdown_latch.h:87
      #13 kudu::server::ServerBase::ExcessLogFileDeleterThread (this=0xffffcf707268) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/server/server_base.cc:721
      #14 0x0000000002029b3c in operator() (this=0x39c67228) at /usr/include/c++/4.8.2/functional:2471
      #15 kudu::Thread::SuperviseThread (arg=0x39c67200) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/util/thread.cc:674
      #16 0x0000ffffb2ec7d38 in start_thread () from /lib64/libpthread.so.0
      #17 0x0000ffffb271f690 in thread_start () from /lib64/libc.so.6

      ...

       

      case 2:

        155  Thread 0xffff93fcef90 (LWP 9) "kudu-tserver" SLL_SetNext (n=0x0, t=0x36e28fc0) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/gperftools-2.6.90/src/linked_list.h:49
        154  Thread 0xffff91ecef90 (LWP 10) "kudu-tserver" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        153  Thread 0xffff916bef90 (LWP 11) "kudu-tserver" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        152  Thread 0xffff90eaef90 (LWP 12) "kernel-watcher-" 0x0000ffff9456cba0 in std::string::_Rep::_S_create(unsigned long, unsigned long, std::allocator<char> const&) () from /lib64/libstdc++.so.6
        151  Thread 0xffff9069ef90 (LWP 13) "sigusr1-handler" 0x0000ffff94ab0708 in sigwait () from /lib64/libpthread.so.0
        150  Thread 0xffff8fe8ef90 (LWP 14) "file cache-evic" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        149  Thread 0xffff8ce2ef90 (LWP 511) "rpc reactor-511" _Function_base (this=0x1191ee450) at /usr/include/c++/4.8.2/functional:2025
        148  Thread 0xffff8b5fef90 (LWP 512) "rpc reactor-512" 0x000000000206b934 in downheap (heap=0xbaae5000, N=240, k=4) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/libev-4.20/ev.c:2141
        147  Thread 0xffff8be0ef90 (LWP 513) "rpc reactor-513" <lambda> (this=<optimized out>) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/consensus/consensus_peers.cc:188
        146  Thread 0xffff8e65ef90 (LWP 514) "rpc reactor-514" 0x0000000000bb0300 in _M_add_ref_lock (this=0x7869f900) at /usr/include/c++/4.8.2/bits/shared_ptr_base.h:246
        145  Thread 0xffff8f25ef90 (LWP 515) "diag-logger-515" 0x0000000001f8a8b4 in sys_futex (u2=0x0, v2=0, t=0xffff8f25e120, v=0, o=128, u=0x43647cf0) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/gutil/linux_syscall_support.h:3401
        144  Thread 0xffff8de4ef90 (LWP 516) "result-tracker-" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        143  Thread 0xffff8d63ef90 (LWP 517) "excess-log-dele" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        142  Thread 0xffff8c61ef90 (LWP 518) "tcmalloc-memory" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        141  Thread 0xffff8adeef90 (LWP 519) "MaintenanceMgr " 0x0000ffff9425543c in sigprocmask () from /lib64/libc.so.6
        140  Thread 0xffff8a5def90 (LWP 520) "MaintenanceMgr " 0x0000ffff942f6aa4 in fdatasync () from /lib64/libc.so.6
        139  Thread 0xffff89dcef90 (LWP 521) "MaintenanceMgr " 0x0000ffff942f6aa4 in fdatasync () from /lib64/libc.so.6
        138  Thread 0xffff895bef90 (LWP 522) "MaintenanceMgr " 0x0000ffff942f6aa4 in fdatasync () from /lib64/libc.so.6
        137  Thread 0xffff88daef90 (LWP 523) "MaintenanceMgr " 0x0000ffff942f6aa4 in fdatasync () from /lib64/libc.so.6
        136  Thread 0xffff7dc4ef90 (LWP 545) "removal_thread-" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        135  Thread 0xffff7d43ef90 (LWP 546) "tc-session-exp-" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        134  Thread 0xffff7cc2ef90 (LWP 547) "rpc worker-547" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        133  Thread 0xffff7c41ef90 (LWP 548) "rpc worker-548" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        132  Thread 0xffff7bc0ef90 (LWP 549) "rpc worker-549" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        131  Thread 0xffff7b3fef90 (LWP 550) "rpc worker-550" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        130  Thread 0xffff7abeef90 (LWP 551) "rpc worker-551" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        129  Thread 0xffff7a3def90 (LWP 552) "rpc worker-552" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        128  Thread 0xffff79bcef90 (LWP 553) "rpc worker-553" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        127  Thread 0xffff793bef90 (LWP 554) "rpc worker-554" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        126  Thread 0xffff78baef90 (LWP 555) "rpc worker-555" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        125  Thread 0xffff7839ef90 (LWP 556) "rpc worker-556" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        124  Thread 0xffff77b8ef90 (LWP 557) "rpc worker-557" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        123  Thread 0xffff7737ef90 (LWP 558) "rpc worker-558" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        122  Thread 0xffff76b6ef90 (LWP 559) "rpc worker-559" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        121  Thread 0xffff7635ef90 (LWP 560) "rpc worker-560" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        120  Thread 0xffff75b4ef90 (LWP 561) "rpc worker-561" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        119  Thread 0xffff7533ef90 (LWP 562) "rpc worker-562" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        118  Thread 0xffff74b2ef90 (LWP 563) "rpc worker-563" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        117  Thread 0xffff7431ef90 (LWP 564) "rpc worker-564" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        116  Thread 0xffff73b0ef90 (LWP 565) "rpc worker-565" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        115  Thread 0xffff732fef90 (LWP 566) "rpc worker-566" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        114  Thread 0xffff72aeef90 (LWP 567) "rpc worker-567" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        113  Thread 0xffff722def90 (LWP 568) "rpc worker-568" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        112  Thread 0xffff71acef90 (LWP 569) "rpc worker-569" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        111  Thread 0xffff70aaef90 (LWP 570) "rpc worker-570" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        110  Thread 0xffff7029ef90 (LWP 572) "rpc worker-572" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        109  Thread 0xffff6fa8ef90 (LWP 573) "rpc worker-573" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        108  Thread 0xffff6f27ef90 (LWP 574) "rpc worker-574" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        107  Thread 0xffff6ea6ef90 (LWP 575) "rpc worker-575" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        106  Thread 0xffff6e25ef90 (LWP 576) "rpc worker-576" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        105  Thread 0xffff6da4ef90 (LWP 577) "rpc worker-577" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        104  Thread 0xffff6d23ef90 (LWP 578) "rpc worker-578" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        103  Thread 0xffff6ca2ef90 (LWP 579) "rpc worker-579" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        102  Thread 0xffff6c21ef90 (LWP 580) "rpc worker-580" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        101  Thread 0xffff6ba0ef90 (LWP 581) "rpc worker-581" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        100  Thread 0xffff6b1fef90 (LWP 582) "rpc worker-582" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        99   Thread 0xffff6a9eef90 (LWP 583) "rpc worker-583" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        98   Thread 0xffff6a1def90 (LWP 584) "rpc worker-584" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        97   Thread 0xffff699cef90 (LWP 585) "rpc worker-585" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        96   Thread 0xffff691bef90 (LWP 586) "rpc worker-586" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        95   Thread 0xffff689aef90 (LWP 587) "rpc worker-587" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        94   Thread 0xffff6819ef90 (LWP 588) "rpc worker-588" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        93   Thread 0xffff6798ef90 (LWP 589) "rpc worker-589" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        92   Thread 0xffff6717ef90 (LWP 590) "rpc worker-590" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        91   Thread 0xffff6696ef90 (LWP 591) "rpc worker-591" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        90   Thread 0xffff6615ef90 (LWP 592) "rpc worker-592" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        89   Thread 0xffff6594ef90 (LWP 593) "rpc worker-593" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        88   Thread 0xffff6513ef90 (LWP 594) "rpc worker-594" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        87   Thread 0xffff6492ef90 (LWP 595) "rpc worker-595" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        86   Thread 0xffff6411ef90 (LWP 596) "rpc worker-596" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        85   Thread 0xffff6390ef90 (LWP 597) "rpc worker-597" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        84   Thread 0xffff630fef90 (LWP 598) "rpc worker-598" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        83   Thread 0xffff628eef90 (LWP 599) "rpc worker-599" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        82   Thread 0xffff620def90 (LWP 600) "rpc worker-600" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        81   Thread 0xffff618cef90 (LWP 601) "rpc worker-601" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        80   Thread 0xffff610bef90 (LWP 602) "rpc worker-602" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        79   Thread 0xffff608aef90 (LWP 603) "rpc worker-603" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        78   Thread 0xffff6009ef90 (LWP 604) "rpc worker-604" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        77   Thread 0xffff5f88ef90 (LWP 605) "rpc worker-605" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        76   Thread 0xffff5f07ef90 (LWP 606) "rpc worker-606" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        75   Thread 0xffff5e86ef90 (LWP 607) "rpc worker-607" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        74   Thread 0xffff5e05ef90 (LWP 608) "rpc worker-608" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        73   Thread 0xffff5d84ef90 (LWP 609) "rpc worker-609" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        72   Thread 0xffff5d03ef90 (LWP 610) "rpc worker-610" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        71   Thread 0xffff5c82ef90 (LWP 611) "rpc worker-611" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        70   Thread 0xffff5c01ef90 (LWP 612) "rpc worker-612" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        69   Thread 0xffff5b80ef90 (LWP 613) "rpc worker-613" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        68   Thread 0xffff5affef90 (LWP 614) "rpc worker-614" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        67   Thread 0xffff5a7eef90 (LWP 615) "rpc worker-615" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        66   Thread 0xffff59fdef90 (LWP 616) "rpc worker-616" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        65   Thread 0xffff597cef90 (LWP 617) "rpc worker-617" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        64   Thread 0xffff58fbef90 (LWP 618) "rpc worker-618" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        63   Thread 0xffff587aef90 (LWP 619) "rpc worker-619" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        62   Thread 0xffff57f9ef90 (LWP 620) "rpc worker-620" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        61   Thread 0xffff5778ef90 (LWP 621) "rpc worker-621" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        60   Thread 0xffff56f7ef90 (LWP 622) "rpc worker-622" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        59   Thread 0xffff5676ef90 (LWP 623) "rpc worker-623" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        58   Thread 0xffff55f5ef90 (LWP 624) "rpc worker-624" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        57   Thread 0xffff5574ef90 (LWP 625) "rpc worker-625" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        56   Thread 0xffff54f3ef90 (LWP 626) "rpc worker-626" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        55   Thread 0xffff5472ef90 (LWP 627) "rpc worker-627" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        54   Thread 0xffff53f1ef90 (LWP 628) "rpc worker-628" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        53   Thread 0xffff5370ef90 (LWP 629) "rpc worker-629" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        52   Thread 0xffff52efef90 (LWP 630) "rpc worker-630" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        51   Thread 0xffff526eef90 (LWP 631) "rpc worker-631" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        50   Thread 0xffff51edef90 (LWP 632) "rpc worker-632" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        49   Thread 0xffff516cef90 (LWP 633) "rpc worker-633" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        48   Thread 0xffff50ebef90 (LWP 634) "rpc worker-634" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        47   Thread 0xffff506aef90 (LWP 635) "rpc worker-635" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        46   Thread 0xffff4fe9ef90 (LWP 636) "rpc worker-636" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        45   Thread 0xffff4f68ef90 (LWP 637) "rpc worker-637" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        44   Thread 0xffff4ee7ef90 (LWP 638) "rpc worker-638" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        43   Thread 0xffff4e66ef90 (LWP 639) "rpc worker-639" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        42   Thread 0xffff4de5ef90 (LWP 640) "rpc worker-640" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        41   Thread 0xffff4d64ef90 (LWP 641) "rpc worker-641" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        40   Thread 0xffff4ce3ef90 (LWP 642) "rpc worker-642" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        39   Thread 0xffff4c62ef90 (LWP 643) "rpc worker-643" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        38   Thread 0xffff4be1ef90 (LWP 644) "rpc worker-644" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        37   Thread 0xffff4b60ef90 (LWP 645) "rpc worker-645" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        36   Thread 0xffff4adfef90 (LWP 646) "rpc worker-646" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        35   Thread 0xffff4a5eef90 (LWP 647) "rpc worker-647" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        34   Thread 0xffff49ddef90 (LWP 648) "acceptor-648" 0x0000ffff94300780 in accept4 () from /lib64/libc.so.6
        33   Thread 0xffff495cef90 (LWP 649) "sq_acceptor" 0x0000ffff942f4e24 in poll () from /lib64/libc.so.6
        32   Thread 0xffff48dbef90 (LWP 650) "heartbeat-650" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        31   Thread 0xffff485aef90 (LWP 651) "heartbeat-651" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        30   Thread 0xffff47d9ef90 (LWP 652) "heartbeat-652" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        29   Thread 0xffff4758ef90 (LWP 653) "maintenance_sch" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        28   Thread 0xffff208bef90 (LWP 1449) "sq_worker" 0x0000ffff942f4e24 in poll () from /lib64/libc.so.6
        27   Thread 0xffff813aef90 (LWP 24789) "raft [worker]-2" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        26   Thread 0xffff7fb7ef90 (LWP 24886) "raft [worker]-2" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        25   Thread 0xffff8038ef90 (LWP 24911) "raft [worker]-2" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        24   Thread 0xffff44d3ef90 (LWP 24912) "raft [worker]-2" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        23   Thread 0xffff424eef90 (LWP 24913) "prepare [worker" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        22   Thread 0xffff42cfef90 (LWP 24914) "wal-append [wor" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        21   Thread 0xffff4350ef90 (LWP 24915) "wal-append [wor" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        20   Thread 0xffff874fef90 (LWP 24916) "prepare [worker" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        19   Thread 0xffff86ceef90 (LWP 24917) "apply [worker]-" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        18   Thread 0xffff864def90 (LWP 24918) "wal-append [wor" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        17   Thread 0xffff85ccef90 (LWP 24919) "apply [worker]-" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        16   Thread 0xffff854bef90 (LWP 24920) "wal-append [wor" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        15   Thread 0xffff84c1ef90 (LWP 24921) "wal-append [wor" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        14   Thread 0xffff8440ef90 (LWP 24922) "wal-append [wor" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        13   Thread 0xffff83bfef90 (LWP 24923) "raft [worker]-2" 0x000000000208358c in sys_wait4 (o=1073741824, r=0x0, s=0xffff83bfb6f4, p=<optimized out>)
          at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/breakpad-9eac2058b70615519b2c4d8c6bdbfca1bd079e39/src/third_party/lss/linux_syscall_support.h:3692
        12   Thread 0xffff833eef90 (LWP 24924) "raft [worker]-2" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        11   Thread 0xffff82bdef90 (LWP 24925) "apply [worker]-" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        10   Thread 0xffff823cef90 (LWP 24926) "wal-append [wor" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        9    Thread 0xffff81bbef90 (LWP 24927) "wal-append [wor" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        8    Thread 0xffff80b9ef90 (LWP 24928) "wal-append [wor" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        7    Thread 0xffff7f36ef90 (LWP 24929) "wal-append [wor" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        6    Thread 0xffff7eb5ef90 (LWP 24930) "wal-append [wor" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        5    Thread 0xffff712bef90 (LWP 24931) "wal-append [wor" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        4    Thread 0xffff46d7ef90 (LWP 24932) "wal-append [wor" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        3    Thread 0xffff4656ef90 (LWP 24933) "wal-append [wor" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        2    Thread 0xffff45d5ef90 (LWP 24935) "wal-append [wor" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0

      • 1    Thread 0xffff94bdb450 (LWP 7) "kudu-tserver" 0x0000ffff94aafcb4 in nanosleep () from /lib64/libpthread.so.0

      ...

      Thread 13 (Thread 0xffff83bfef90 (LWP 24923)):
      #0  0x000000000208358c in sys_wait4 (o=1073741824, r=0x0, s=0xffff83bfb6f4, p=<optimized out>) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/breakpad-9eac2058b70615519b2c4d8c6bdbfca1bd079e39/src/third_party/lss/linux_syscall_support.h:3692
      #1  sys_waitpid (options=1073741824, status=0xffff83bfb6f4, pid=<optimized out>) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/breakpad-9eac2058b70615519b2c4d8c6bdbfca1bd079e39/src/third_party/lss/linux_syscall_support.h:3697
      #2  google_breakpad::ExceptionHandler::GenerateDump (this=0x1a03e4d0, context=<optimized out>) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/breakpad-9eac2058b70615519b2c4d8c6bdbfca1bd079e39/src/client/linux/handler/exception_handler.cc:553
      #3  0x00000000020838fc in google_breakpad::ExceptionHandler::SignalHandler (sig=11, info=0xffff83bfb810, uc=0xffff83bfb890)
          at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/breakpad-9eac2058b70615519b2c4d8c6bdbfca1bd079e39/src/client/linux/handler/exception_handler.cc:376
      #4  <signal handler called>
      #5  access_mem (as=0x2dc6220 <local_addr_space>, addr=1, val=0xffff83bfca98, write=0, arg=0xffff83bfcb40) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/libunwind-1.4.0/src/aarch64/Ginit.c:96
      #6  0x0000000002066b6c in _ULaarch64_is_signal_frame (cursor=cursor@entry=0xffff83bfce20) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/libunwind-1.4.0/src/aarch64/Gis_signal_frame.c:51
      #7  0x0000000002063d18 in _ULaarch64_step (cursor=0xffff83bfce20) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/libunwind-1.4.0/src/aarch64/Gstep.c:143
      #8  0x0000000001f88fc0 in kudu::StackTrace::Collect (this=this@entry=0x38073978, skip_frames=0, skip_frames@entry=1) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/util/debug-util.cc:612
      #9  0x0000000001f8948c in Collect (skip_frames=1, this=0x38073978) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/util/debug-util.cc:288
      #10 kudu::(anonymous namespace)::HandleStackTraceSignal (info=<optimized out>) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/util/debug-util.cc:286
      #11 <signal handler called>
      #12 0x0000ffff942ff640 in clone () from /lib64/libc.so.6
      #13 0x0000ffff94aa6d78 in do_clone.constprop.3 () from /lib64/libpthread.so.0
      Backtrace stopped: previous frame inner to this frame (corrupt stack?)

       

      Thread 141 (Thread 0xffff8adeef90 (LWP 519)):
      #0  0x0000ffff9425543c in sigprocmask () from /lib64/libc.so.6
      #1  0x0000000002065dc8 in put_rs_cache (as=<optimized out>, saved_maskp=0xffff8adead70, cache=0x2dc6280 <local_addr_space+96>) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/libunwind-1.4.0/src/dwarf/Gparser.c:634
      #2  find_reg_state (sr=0xffff8adeadf0, c=0xffff8adeb890) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/libunwind-1.4.0/src/dwarf/Gparser.c:949
      #3  _ULaarch64_dwarf_step (c=c@entry=0xffff8adeb890) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/libunwind-1.4.0/src/dwarf/Gparser.c:965
      #4  0x0000000002063ef4 in _ULaarch64_step (cursor=0xffff8adeb890) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/libunwind-1.4.0/src/aarch64/Gstep.c:146
      #5  0x0000000001f88f6c in kudu::StackTrace::Collect (this=this@entry=0x3806e978, skip_frames=skip_frames@entry=1) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/util/debug-util.cc:612
      #6  0x0000000001f8948c in Collect (skip_frames=1, this=0x3806e978) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/util/debug-util.cc:288
      #7  kudu::(anonymous namespace)::HandleStackTraceSignal (info=<optimized out>) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/util/debug-util.cc:286
      #8  <signal handler called>
      #9  0x0000ffff942f6aa4 in fdatasync () from /lib64/libc.so.6
      #10 0x0000000001fa9dec in kudu::(anonymous namespace)::DoSync (fd=4614, filename="/opt/java/kudu-1.13.0-aarch64/kudu-1.13.0-aarch64/tserver/data/7/data/80eb86ffb4af46b397440586d926cce8.data")
          at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/util/env_posix.cc:346
      #11 0x0000000001faa814 in kudu::(anonymous namespace)::PosixRWFile::Sync (this=0x1e03b580) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/util/env_posix.cc:983
      #12 0x0000000001fba228 in kudu::internal::Descriptor<kudu::RWFile>::Sync (this=<optimized out>) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/util/file_cache.cc:293
      #13 0x0000000001daae1c in kudu::fs::internal::LogBlockContainer::SyncData (this=0x1d9a0dc0) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/fs/log_block_manager.cc:1213
      #14 0x0000000001db46bc in kudu::fs::internal::LogBlockContainer::{}lambda8::operator() ({}closure={_}_closure@entry=0xffff8aded768) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/fs/log_block_manager.cc:1105
      #15 0x0000000001db516c in DoCloseBlocks (mode=kudu::fs::internal::LogBlockContainer::SYNC, blocks=std::vector of length 1, capacity 1 = {...}, this=<optimized out>, this=<optimized out>)
          at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/fs/log_block_manager.cc:1129
      #16 kudu::fs::internal::LogBlockCreationTransaction::CommitCreatedBlocks (this=0x4bb325a0) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/fs/log_block_manager.cc:1416
      #17 0x0000000000b402c0 in kudu::tablet::RollingDiskRowSetWriter::Finish (this=this@entry=0xffff8adedde8) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/tablet/diskrowset.cc:487
      #18 0x0000000000af8f04 in kudu::tablet::Tablet::DoMergeCompactionOrFlush (this=this@entry=0x7d614a80, input=..., mrs_being_flushed=mrs_being_flushed@entry=-1) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/tablet/tablet.cc:1608
      #19 0x0000000000afb6d0 in kudu::tablet::Tablet::Compact (this=0x7d614a80, flags=flags@entry=0) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/tablet/tablet.cc:1858
      #20 0x0000000000b1263c in kudu::tablet::CompactRowSetsOp::Perform (this=0x1b3067e0) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/tablet/tablet_mm_ops.cc:176
      #21 0x0000000001fd9060 in kudu::MaintenanceManager::LaunchOp (this=0x2153f878, op=0x1b3067e0) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/util/maintenance_manager.cc:545
      #22 0x0000000002031798 in operator() (this=0xffff8adee628) at /usr/include/c++/4.8.2/functional:2471
      #23 kudu::ThreadPool::DispatchThread (this=0x1f9345a0) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/util/threadpool.cc:662
      #24 0x0000000002029b3c in operator() (this=0x1ce37768) at /usr/include/c++/4.8.2/functional:2471
      #25 kudu::Thread::SuperviseThread (arg=0x1ce37740) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/util/thread.cc:674
      #26 0x0000ffff94aa7d38 in start_thread () from /lib64/libpthread.so.0
      #27 0x0000ffff942ff690 in thread_start () from /lib64/libc.so.6

       

      case 3:

        178  Thread 0xffffb081ef90 (LWP 9) "kudu-tserver" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        177  Thread 0xffffae71ef90 (LWP 10) "kudu-tserver" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        176  Thread 0xffffadf0ef90 (LWP 11) "kudu-tserver" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        175  Thread 0xffffad6fef90 (LWP 12) "kernel-watcher-" 0x0000ffffb0ae8898 in {}GI{_}_IO_link_in () from /lib64/libc.so.6
        174  Thread 0xffffaceeef90 (LWP 13) "sigusr1-handler" 0x0000ffffb1300708 in sigwait () from /lib64/libpthread.so.0
        173  Thread 0xffffac6def90 (LWP 14) "file cache-evic" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        172  Thread 0xffff9b4bef90 (LWP 648) "rpc reactor-648" __weak_ptr (this=<optimized out>) at /usr/include/c++/4.8.2/bits/shared_ptr_base.h:1195
        171  Thread 0xffff8caeef90 (LWP 649) "rpc reactor-649" _Hash_node_base (this=<optimized out>) at /usr/include/c++/4.8.2/bits/hashtable_policy.h:153
        170  Thread 0xffff94beef90 (LWP 650) "rpc reactor-650" _Function_base (this=0x2eeb63c0) at /usr/include/c++/4.8.2/functional:2025
        169  Thread 0xffffa8e6ef90 (LWP 651) "rpc reactor-651" 0x0000ffffb0b65030 in clock_gettime () from /lib64/libc.so.6
        168  Thread 0xffffabaaef90 (LWP 652) "diag-logger-652" 0x0000000001f8a8b4 in sys_futex (u2=0x0, v2=0, t=0xffffabaae120, v=0, o=128, u=0x191ad350) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/gutil/linux_syscall_support.h:3401
        167  Thread 0xffffab29ef90 (LWP 653) "result-tracker-" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        166  Thread 0xffffaaa8ef90 (LWP 654) "excess-log-dele" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        165  Thread 0xffffaa27ef90 (LWP 655) "tcmalloc-memory" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        164  Thread 0xffffa9a6ef90 (LWP 656) "MaintenanceMgr " 0x0000ffffb0b46aa4 in fdatasync () from /lib64/libc.so.6
        163  Thread 0xffff9dd0ef90 (LWP 679) "removal_thread-" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        162  Thread 0xffff9d4fef90 (LWP 680) "tc-session-exp-" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        161  Thread 0xffff9cceef90 (LWP 681) "rpc worker-681" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        160  Thread 0xffff9c4def90 (LWP 682) "rpc worker-682" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        159  Thread 0xffff9bccef90 (LWP 683) "rpc worker-683" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        158  Thread 0xffff9acaef90 (LWP 684) "rpc worker-684" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        157  Thread 0xffff9a49ef90 (LWP 685) "rpc worker-685" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        156  Thread 0xffff99c8ef90 (LWP 686) "rpc worker-686" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        155  Thread 0xffff9947ef90 (LWP 687) "rpc worker-687" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        154  Thread 0xffff98c6ef90 (LWP 688) "rpc worker-688" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        153  Thread 0xffff9845ef90 (LWP 689) "rpc worker-689" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        152  Thread 0xffff97c4ef90 (LWP 690) "rpc worker-690" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        151  Thread 0xffff96c2ef90 (LWP 691) "rpc worker-691" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        150  Thread 0xffff9641ef90 (LWP 692) "rpc worker-692" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        149  Thread 0xffff95c0ef90 (LWP 693) "rpc worker-693" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        148  Thread 0xffff953fef90 (LWP 694) "rpc worker-694" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        147  Thread 0xffff943def90 (LWP 695) "rpc worker-695" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        146  Thread 0xffff93bcef90 (LWP 696) "rpc worker-696" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        145  Thread 0xffff933bef90 (LWP 697) "rpc worker-697" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        144  Thread 0xffff92baef90 (LWP 698) "rpc worker-698" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        143  Thread 0xffff9239ef90 (LWP 699) "rpc worker-699" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        142  Thread 0xffff91b8ef90 (LWP 700) "rpc worker-700" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        141  Thread 0xffff9137ef90 (LWP 701) "rpc worker-701" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        140  Thread 0xffff90b6ef90 (LWP 702) "rpc worker-702" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        139  Thread 0xffff9035ef90 (LWP 703) "rpc worker-703" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        138  Thread 0xffff8fb4ef90 (LWP 704) "rpc worker-704" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        137  Thread 0xffff8f33ef90 (LWP 705) "rpc worker-705" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        136  Thread 0xffff8eb2ef90 (LWP 706) "rpc worker-706" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        135  Thread 0xffff8e31ef90 (LWP 707) "rpc worker-707" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        134  Thread 0xffff8db0ef90 (LWP 708) "rpc worker-708" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        133  Thread 0xffff8d2fef90 (LWP 709) "rpc worker-709" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        132  Thread 0xffff8c2def90 (LWP 710) "rpc worker-710" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        131  Thread 0xffff8bacef90 (LWP 711) "rpc worker-711" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        130  Thread 0xffff8b2bef90 (LWP 712) "rpc worker-712" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        129  Thread 0xffff8aaaef90 (LWP 713) "rpc worker-713" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        128  Thread 0xffff8a29ef90 (LWP 714) "rpc worker-714" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        127  Thread 0xffff89a8ef90 (LWP 715) "rpc worker-715" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        126  Thread 0xffff8927ef90 (LWP 716) "rpc worker-716" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        125  Thread 0xffff88a6ef90 (LWP 717) "rpc worker-717" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        124  Thread 0xffff8825ef90 (LWP 718) "rpc worker-718" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        123  Thread 0xffff87a4ef90 (LWP 719) "rpc worker-719" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        122  Thread 0xffff8723ef90 (LWP 720) "rpc worker-720" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        121  Thread 0xffff86a2ef90 (LWP 721) "rpc worker-721" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        120  Thread 0xffff8621ef90 (LWP 722) "rpc worker-722" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        119  Thread 0xffff85a0ef90 (LWP 723) "rpc worker-723" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        118  Thread 0xffff851fef90 (LWP 724) "rpc worker-724" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        117  Thread 0xffff849eef90 (LWP 725) "rpc worker-725" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        116  Thread 0xffff841def90 (LWP 726) "rpc worker-726" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        115  Thread 0xffff839cef90 (LWP 727) "rpc worker-727" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        114  Thread 0xffff831bef90 (LWP 728) "rpc worker-728" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        113  Thread 0xffff829aef90 (LWP 729) "rpc worker-729" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        112  Thread 0xffff8219ef90 (LWP 730) "rpc worker-730" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        111  Thread 0xffff8198ef90 (LWP 731) "rpc worker-731" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        110  Thread 0xffff8117ef90 (LWP 732) "rpc worker-732" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        109  Thread 0xffff8096ef90 (LWP 733) "rpc worker-733" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        108  Thread 0xffff8015ef90 (LWP 734) "rpc worker-734" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        107  Thread 0xffff7f94ef90 (LWP 735) "rpc worker-735" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        106  Thread 0xffff7f13ef90 (LWP 736) "rpc worker-736" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        105  Thread 0xffff7e92ef90 (LWP 737) "rpc worker-737" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        104  Thread 0xffff7e11ef90 (LWP 738) "rpc worker-738" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        103  Thread 0xffff7d90ef90 (LWP 739) "rpc worker-739" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        102  Thread 0xffff7d0fef90 (LWP 740) "rpc worker-740" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        101  Thread 0xffff7c8eef90 (LWP 741) "rpc worker-741" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        100  Thread 0xffff7c0def90 (LWP 742) "rpc worker-742" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        99   Thread 0xffff7b8cef90 (LWP 743) "rpc worker-743" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        98   Thread 0xffff7b0bef90 (LWP 744) "rpc worker-744" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        97   Thread 0xffff7a8aef90 (LWP 745) "rpc worker-745" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        96   Thread 0xffff7a09ef90 (LWP 746) "rpc worker-746" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        95   Thread 0xffff7988ef90 (LWP 747) "rpc worker-747" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        94   Thread 0xffff7907ef90 (LWP 748) "rpc worker-748" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        93   Thread 0xffff7886ef90 (LWP 749) "rpc worker-749" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        92   Thread 0xffff7805ef90 (LWP 750) "rpc worker-750" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        91   Thread 0xffff7784ef90 (LWP 751) "rpc worker-751" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        90   Thread 0xffff7703ef90 (LWP 752) "rpc worker-752" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        89   Thread 0xffff7682ef90 (LWP 753) "rpc worker-753" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        88   Thread 0xffff7601ef90 (LWP 754) "rpc worker-754" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        87   Thread 0xffff7580ef90 (LWP 755) "rpc worker-755" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        86   Thread 0xffff74ffef90 (LWP 756) "rpc worker-756" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        85   Thread 0xffff747eef90 (LWP 757) "rpc worker-757" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        84   Thread 0xffff73fdef90 (LWP 758) "rpc worker-758" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        83   Thread 0xffff737cef90 (LWP 759) "rpc worker-759" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        82   Thread 0xffff72fbef90 (LWP 760) "rpc worker-760" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        81   Thread 0xffff727aef90 (LWP 761) "rpc worker-761" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        80   Thread 0xffff71f9ef90 (LWP 762) "rpc worker-762" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        79   Thread 0xffff7178ef90 (LWP 763) "rpc worker-763" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        78   Thread 0xffff70f7ef90 (LWP 764) "rpc worker-764" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        77   Thread 0xffff7076ef90 (LWP 765) "rpc worker-765" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        76   Thread 0xffff6ff5ef90 (LWP 766) "rpc worker-766" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        75   Thread 0xffff6f74ef90 (LWP 767) "rpc worker-767" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        74   Thread 0xffff6ef3ef90 (LWP 768) "rpc worker-768" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        73   Thread 0xffff6e72ef90 (LWP 769) "rpc worker-769" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        72   Thread 0xffff6df1ef90 (LWP 770) "rpc worker-770" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        71   Thread 0xffff6d70ef90 (LWP 771) "rpc worker-771" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        70   Thread 0xffff6cefef90 (LWP 772) "rpc worker-772" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        69   Thread 0xffff6c6eef90 (LWP 773) "rpc worker-773" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        68   Thread 0xffff6bedef90 (LWP 774) "rpc worker-774" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        67   Thread 0xffff6b6cef90 (LWP 775) "rpc worker-775" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        66   Thread 0xffff6aebef90 (LWP 776) "rpc worker-776" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        65   Thread 0xffff6a6aef90 (LWP 777) "rpc worker-777" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        64   Thread 0xffff69e9ef90 (LWP 778) "rpc worker-778" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        63   Thread 0xffff6968ef90 (LWP 779) "rpc worker-779" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        62   Thread 0xffff68e7ef90 (LWP 780) "rpc worker-780" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        61   Thread 0xffff6866ef90 (LWP 781) "acceptor-781" 0x0000ffffb0b50780 in accept4 () from /lib64/libc.so.6
        60   Thread 0xffff67e5ef90 (LWP 782) "sq_acceptor" 0x0000ffffb0b44e24 in poll () from /lib64/libc.so.6
        59   Thread 0xffff6764ef90 (LWP 783) "heartbeat-783" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        58   Thread 0xffff66e3ef90 (LWP 784) "heartbeat-784" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        57   Thread 0xffff6662ef90 (LWP 785) "heartbeat-785" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        56   Thread 0xffff65e1ef90 (LWP 786) "maintenance_sch" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        55   Thread 0xffff51b9ef90 (LWP 1000) "sq_worker" 0x0000ffffb0b44e24 in poll () from /lib64/libc.so.6
        54   Thread 0xffff9fb4ef90 (LWP 10412) "raft [worker]-1" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        53   Thread 0xffff5b4cef90 (LWP 10567) "raft [worker]-1" 0x0000ffffb12fefcc in __lll_lock_wait () from /lib64/libpthread.so.0
        52   Thread 0xffff4e32ef90 (LWP 10592) "raft [worker]-1" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        51   Thread 0xffff5d50ef90 (LWP 10599) "raft [worker]-1" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        50   Thread 0xffff4db1ef90 (LWP 10610) "prepare [worker" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        49   Thread 0xffff4d30ef90 (LWP 10609) "prepare [worker" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        48   Thread 0xffff4fb5ef90 (LWP 10611) "prepare [worker" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        47   Thread 0xffff4cafef90 (LWP 10612) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        46   Thread 0xffffa6c7ef90 (LWP 10613) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        45   Thread 0xffffa646ef90 (LWP 10614) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        44   Thread 0xffffa5c5ef90 (LWP 10615) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        43   Thread 0xffffa544ef90 (LWP 10616) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        42   Thread 0xffffa4c3ef90 (LWP 10617) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        41   Thread 0xffffa442ef90 (LWP 10618) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        40   Thread 0xffffa3c1ef90 (LWP 10619) "apply [worker]-" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        39   Thread 0xffffa340ef90 (LWP 10620) "apply [worker]-" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        38   Thread 0xffffa2bfef90 (LWP 10621) "apply [worker]-" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        37   Thread 0xffffa23eef90 (LWP 10622) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        36   Thread 0xffffa1bdef90 (LWP 10623) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        35   Thread 0xffffa0b6ef90 (LWP 10624) "raft [worker]-1" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        34   Thread 0xffffa035ef90 (LWP 10625) "raft [worker]-1" 0x00000000020c7d84 in store (__m=std::memory_order_relaxed, __i=40, this=0x27a358f74) at /usr/include/c++/4.8.2/bits/atomic_base.h:474
        33   Thread 0xffff9f33ef90 (LWP 10626) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        32   Thread 0xffff9743ef90 (LWP 10627) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        31   Thread 0xffff6560ef90 (LWP 10628) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        30   Thread 0xffff64dfef90 (LWP 10629) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        29   Thread 0xffff645eef90 (LWP 10630) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        28   Thread 0xffff63ddef90 (LWP 10631) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        27   Thread 0xffff635cef90 (LWP 10632) "rpc worker-735" 0x000000000208358c in sys_wait4 (o=1073741824, r=0x0, s=0xffff635cab74, p=<optimized out>)
          at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/breakpad-9eac2058b70615519b2c4d8c6bdbfca1bd079e39/src/third_party/lss/linux_syscall_support.h:3692
        26   Thread 0xffff62dbef90 (LWP 10633) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        25   Thread 0xffff625aef90 (LWP 10634) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        24   Thread 0xffff61d9ef90 (LWP 10635) "apply [worker]-" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        23   Thread 0xffff6158ef90 (LWP 10636) "apply [worker]-" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        22   Thread 0xffff60d7ef90 (LWP 10637) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        21   Thread 0xffff6056ef90 (LWP 10638) "prepare [worker" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        20   Thread 0xffff5fd5ef90 (LWP 10639) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        19   Thread 0xffff5f54ef90 (LWP 10640) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        18   Thread 0xffff5ed3ef90 (LWP 10641) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        17   Thread 0xffff5e52ef90 (LWP 10642) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        16   Thread 0xffff5dd1ef90 (LWP 10643) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        15   Thread 0xffff5ccfef90 (LWP 10644) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        14   Thread 0xffff5c4eef90 (LWP 10645) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        13   Thread 0xffff5bcdef90 (LWP 10646) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        12   Thread 0xffff5acbef90 (LWP 10647) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        11   Thread 0xffff5a4aef90 (LWP 10648) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        10   Thread 0xffff59c9ef90 (LWP 10649) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        9    Thread 0xffff5948ef90 (LWP 10650) "apply [worker]-" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        8    Thread 0xffff58c7ef90 (LWP 10651) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        7    Thread 0xffff5846ef90 (LWP 10652) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        6    Thread 0xffff57c5ef90 (LWP 10653) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        5    Thread 0xffff5744ef90 (LWP 10654) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        4    Thread 0xffff56c3ef90 (LWP 10655) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        3    Thread 0xffff5642ef90 (LWP 10656) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
        2    Thread 0xffff55c1ef90 (LWP 10657) "wal-append [wor" 0x00000000020bbffc in kudu::consensus::OpId::~OpId (this=0x1f8b90198, __in_chrg=<optimized out>) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/build/release/src/kudu/consensus/opid.pb.cc:126

      • 1    Thread 0xffffb142b450 (LWP 7) "kudu-tserver" 0x0000ffffb12ffcb4 in nanosleep () from /lib64/libpthread.so.0

      ...

      Thread 27 (Thread 0xffff635cef90 (LWP 10632)):
      #0  0x000000000208358c in sys_wait4 (o=1073741824, r=0x0, s=0xffff635cab74, p=<optimized out>) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/breakpad-9eac2058b70615519b2c4d8c6bdbfca1bd079e39/src/third_party/lss/linux_syscall_support.h:3692
      #1  sys_waitpid (options=1073741824, status=0xffff635cab74, pid=<optimized out>) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/breakpad-9eac2058b70615519b2c4d8c6bdbfca1bd079e39/src/third_party/lss/linux_syscall_support.h:3697
      #2  google_breakpad::ExceptionHandler::GenerateDump (this=0x12d0c4d0, context=<optimized out>) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/breakpad-9eac2058b70615519b2c4d8c6bdbfca1bd079e39/src/client/linux/handler/exception_handler.cc:553
      #3  0x00000000020838fc in google_breakpad::ExceptionHandler::SignalHandler (sig=11, info=0xffff635cac90, uc=0xffff635cad10)
          at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/breakpad-9eac2058b70615519b2c4d8c6bdbfca1bd079e39/src/client/linux/handler/exception_handler.cc:376
      #4  <signal handler called>
      #5  dwarf_get (c=0xffff635cce20, val=<synthetic pointer>, loc=...) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/libunwind-1.4.0/include/tdep-aarch64/libunwind_i.h:146
      #6  apply_reg_state (c=c@entry=0xffff635cce20, rs=rs@entry=0xffff635cc718) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/libunwind-1.4.0/src/dwarf/Gparser.c:878
      #7  0x0000000002065dd8 in _ULaarch64_dwarf_step (c=c@entry=0xffff635cce20) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/libunwind-1.4.0/src/dwarf/Gparser.c:967
      #8  0x0000000002063ef4 in _ULaarch64_step (cursor=0xffff635cce20) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/libunwind-1.4.0/src/aarch64/Gstep.c:146
      #9  0x0000000001f88fc0 in kudu::StackTrace::Collect (this=this@entry=0x75d33f18, skip_frames=0, skip_frames@entry=1) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/util/debug-util.cc:612
      #10 0x0000000001f8948c in Collect (skip_frames=1, this=0x75d33f18) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/util/debug-util.cc:288
      #11 kudu::(anonymous namespace)::HandleStackTraceSignal (info=<optimized out>) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/util/debug-util.cc:286
      #12 <signal handler called>
      #13 0x0000ffffb0b4f640 in clone () from /lib64/libc.so.6
      #14 0x0000ffffb12f6d78 in do_clone.constprop.3 () from /lib64/libpthread.so.0
      #15 0x0000ffffb12f84f8 in pthread_create@@GLIBC_2.17 () from /lib64/libpthread.so.0
      #16 0x00000000020268bc in kudu::Thread::StartThread(std::string, std::string, std::function<void ()>, unsigned long, scoped_refptr<kudu::Thread>*) (category=..., name=..., functor=..., flags=<error reading variable: Cannot access memory at address 0x69>, 
          holder=<error reading variable: Cannot access memory at address 0x89>) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/util/thread.cc:618
      Backtrace stopped: previous frame inner to this frame (corrupt stack?)

       

      Attachments

        Activity

          People

            Unassigned Unassigned
            motto1314 Jie Huang
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated: