[TRAFODION-1107] LP Bug: 1438466 - Multiple tdm_arkcmp child processes started after receipt of HBase error - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Open
Priority: Critical
Resolution: Unresolved
Affects Version/s: None
Fix Version/s: None
Component/s: sql-cmp
Labels:
None

External issue URL:
https://bugs.launchpad.net/bugs/1438466

Description

During a fresh test of running the compGeneral regression suite
while artificially interjecting an error return from the TrxRegionEndpoint
coprocessor, numerous tdm_arkcmp child processes were started.

Before the error hit, we seemed to have a normal number of compilers

[$Z0005MG] 000,00003170 001 GEN ES-A- $Z0002KK $Z0002IV tdm_arkcmp
[$Z0005MG] 000,00003292 001 GEN ES-A- $Z0002P2 $Z0002KK tdm_arkcmp
[$Z0005MG] 000,00003816 001 GEN ES-A- $Z000341 $Z0002P2 tdm_arkcmp
[$Z0005MG] 000,00003886 001 GEN ES-A- $Z000361 $Z000341 tdm_arkcmp

After forcing the error, it looks like we have new compilers being generated,
all ultimately part of the original tdm_arkcmp parent off of the sqlci session.

This is a result of a drop statement. From the sqlci window, the
statement appears hung, as it never returns. But, it appears the
compilers keep generating new children and the query ultimately never returns.

When I killed the query, it had 174 compilers running.

I tried a pstack for one of the compilers, I’ve attached it below.

g4t3037

{joaniec}3: sqps
Processing cluster.conf on local host g4t3037.houston.hp.com
[$Z000AF9] Shell/shell Version 1.0.1 Release 1.1.0 (Build release [joaniec], date 26Mar15)
[$Z000AF9] %ps
[$Z000AF9] NID,PID(os) PRI TYPE STATES NAME PARENT PROGRAM
[$Z000AF9] ------------ — ---- ------- ----------- ----------- ---------------
[$Z000AF9] 000,00018562 000 WDG ES-A- $WDG000 NONE sqwatchdog
[$Z000AF9] 000,00018563 000 PSD ES-A- $PSD000 NONE pstartd
[$Z000AF9] 000,00018592 001 DTM ES-A- $TM0 NONE tm
[$Z000AF9] 000,00019243 001 GEN ES-A- $ZSC000 NONE mxsscp
[$Z000AF9] 000,00019274 001 SSMP ES-A- $ZSM000 NONE mxssmp
[$Z000AF9] 000,00020982 001 GEN ES-A- $ZLOBSRV0 NONE mxlobsrvr
[$Z000AF9] 000,00007356 001 GEN ES-A- $Z000606 NONE sqlci
[$Z000AF9] 000,00007416 001 GEN ES-A- $Z00061W $Z000606 tdm_arkcmp
[$Z000AF9] 000,00007960 001 GEN ES-A- $Z0006HF $Z00061W tdm_arkcmp
[$Z000AF9] 000,00008021 001 GEN ES-A- $Z0006J6 $Z0006HF tdm_arkcmp
[$Z000AF9] 000,00008079 001 GEN ES-A- $Z0006KU $Z0006J6 tdm_arkcmp
[$Z000AF9] 000,00008137 001 GEN ES-A- $Z0006MH $Z0006KU tdm_arkcmp
[$Z000AF9] 000,00008194 001 GEN ES-A- $Z0006P4 $Z0006MH tdm_arkcmp
[$Z000AF9] 000,00008252 001 GEN ES-A- $Z0006QS $Z0006P4 tdm_arkcmp
[$Z000AF9] 000,00008312 001 GEN ES-A- $Z0006SH $Z0006QS tdm_arkcmp
[$Z000AF9] 000,00008369 001 GEN ES-A- $Z0006U4 $Z0006SH tdm_arkcmp
[$Z000AF9] 000,00008427 001 GEN ES-A- $Z0006VS $Z0006U4 tdm_arkcmp
[$Z000AF9] 000,00008491 001 GEN ES-A- $Z0006XL $Z0006VS tdm_arkcmp
[$Z000AF9] 000,00009023 001 GEN ES-A- $Z0007CT $Z0006XL tdm_arkcmp
[$Z000AF9] 000,00009081 001 GEN ES-A- $Z0007EG $Z0007CT tdm_arkcmp
[$Z000AF9] 000,00009141 001 GEN ES-A- $Z0007G6 $Z0007EG tdm_arkcmp
[$Z000AF9] 000,00009202 001 GEN ES-A- $Z0007HX $Z0007G6 tdm_arkcmp
[$Z000AF9] 000,00009262 001 GEN ES-A- $Z0007JM $Z0007HX tdm_arkcmp
[$Z000AF9] 000,00009320 001 GEN ES-A- $Z0007LA $Z0007JM tdm_arkcmp
[$Z000AF9] 000,00009489 001 GEN ES-A- $Z0007R4 $Z0007LA tdm_arkcmp
[$Z000AF9] 000,00009547 001 GEN ES-A- $Z0007SS $Z0007R4 tdm_arkcmp
[$Z000AF9] 000,00009604 001 GEN ES-A- $Z0007UE $Z0007SS tdm_arkcmp
[$Z000AF9] 000,00009661 001 GEN ES-A- $Z0007W1 $Z0007UE tdm_arkcmp
[$Z000AF9] 000,00009728 001 GEN ES-A- $Z0007XY $Z0007W1 tdm_arkcmp
[$Z000AF9] 000,00010268 001 GEN ES-A- $Z0008DD $Z0007XY tdm_arkcmp
[$Z000AF9] 000,00010364 001 GEN ES-A- $Z0008G4 $Z0008DD tdm_arkcmp
[$Z000AF9] 000,00010421 001 GEN ES-A- $Z0008HR $Z0008G4 tdm_arkcmp
[$Z000AF9] 000,00010479 001 GEN ES-A- $Z0008JE $Z0008HR tdm_arkcmp
[$Z000AF9] 000,00010537 001 GEN ES-A- $Z0008L2 $Z0008JE tdm_arkcmp
[$Z000AF9] 000,00010620 001 GEN ES-A- $Z0008NF $Z0008L2 tdm_arkcmp
[$Z000AF9] 000,00010677 001 GEN ES-A- $Z0008Q2 $Z0008NF tdm_arkcmp
[$Z000AF9] 000,00010735 001 GEN ES-A- $Z0008RQ $Z0008Q2 tdm_arkcmp
[$Z000AF9] 000,00010796 001 GEN ES-A- $Z0008TG $Z0008RQ tdm_arkcmp
[$Z000AF9] 000,00010857 001 GEN ES-A- $Z0008V7 $Z0008TG tdm_arkcmp
[$Z000AF9] 000,00011367 001 GEN ES-A- $Z00099S $Z0008V7 tdm_arkcmp
[$Z000AF9] 000,00011437 001 GEN ES-A- $Z0009BS $Z00099S tdm_arkcmp
[$Z000AF9] 000,00011496 001 GEN ES-A- $Z0009DG $Z0009BS tdm_arkcmp
[$Z000AF9] 000,00011558 001 GEN ES-A- $Z0009F8 $Z0009DG tdm_arkcmp
[$Z000AF9] 000,00011618 001 GEN ES-A- $Z0009GY $Z0009F8 tdm_arkcmp
[$Z000AF9] 000,00011680 001 GEN ES-A- $Z0009IQ $Z0009GY tdm_arkcmp
[$Z000AF9] 000,00011741 001 GEN ES-A- $Z0009KG $Z0009IQ tdm_arkcmp
[$Z000AF9] 000,00011801 001 GEN ES-A- $Z0009M6 $Z0009KG tdm_arkcmp
[$Z000AF9] 000,00011862 001 GEN ES-A- $Z0009NX $Z0009M6 tdm_arkcmp
[$Z000AF9] 000,00011921 001 GEN ES-A- $Z0009QL $Z0009NX tdm_arkcmp
[$Z000AF9] 000,00012410 001 GEN ES-A- $Z000A4K $Z0009QL tdm_arkcmp
[$Z000AF9] 000,00012509 001 GEN ES-A- $Z000A7E $Z000A4K tdm_arkcmp
[$Z000AF9] 000,00012582 001 GEN ES-A- $Z000A9H $Z000A7E tdm_arkcmp
[$Z000AF9] 000,00012646 001 GEN ES-A- $Z000ABB $Z000A9H tdm_arkcmp
[$Z000AF9] 000,00012703 001 GEN ES-A- $Z000ACY $Z000ABB tdm_arkcmp
[$Z000AF9] 000,00012760 001 GEN ES-A- $Z000AEK $Z000ACY tdm_arkcmp
[$Z000AF9] 000,00012784 001 GEN ES-A- $Z000AF9 NONE shell
[$Z000AF9] 001,00018560 000 WDG ES-A- $WDG001 NONE sqwatchdog
[$Z000AF9] 001,00018561 000 PSD ES-A- $PSD001 NONE pstartd
[$Z000AF9] 001,00018640 001 DTM ES-A- $TM1 NONE tm
[$Z000AF9] 001,00019258 001 GEN ES-A- $ZSC001 NONE mxsscp
[$Z000AF9] 001,00019289 001 SSMP ES-A- $ZSM001 NONE mxssmp
[$Z000AF9] 001,00020989 001 GEN ES-A- $ZLOBSRV1 NONE mxlobsrvr

[$Z000GHJ] 000,00020170 001 GEN ES-A- $Z000GGA $Z000GEK tdm_arkcmp
[$Z000GHJ] 000,00020214 001 GEN ES-A- $Z000GHJ NONE shell
[$Z000GHJ] 001,00018560 000 WDG ES-A- $WDG001 NONE sqwatchdog
[$Z000GHJ] 001,00018561 000 PSD ES-A- $PSD001 NONE pstartd
[$Z000GHJ] 001,00018640 001 DTM ES-A- $TM1 NONE tm
[$Z000GHJ] 001,00019258 001 GEN ES-A- $ZSC001 NONE mxsscp
[$Z000GHJ] 001,00019289 001 SSMP ES-A- $ZSM001 NONE mxssmp
[$Z000GHJ] 001,00020989 001 GEN ES-A- $ZLOBSRV1 NONE mxlobsrvr

g4t3037{joaniec}

120: pstack $Z000GGA
Usage: pstack <process-id>
g4t3037

{joaniec}

121: pstack 20110
Thread 30 (Thread 0x7fffee0a7700 (LWP 20111)):
#0 0x0000003e3c00e84d in accept () from /lib64/libpthread.so.0
#1 0x00007fffefec1995 in SB_Trans::Sock_Listener::accept() () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbms.so
#2 0x00007fffefec84e6 in SB_Trans::Sock_Stream_Accept_Thread::run() () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbms.so
#3 0x00007fffefec8213 in sock_stream_accept_thread_fun(void*) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbms.so
#4 0x00007fffee540b4f in SB_Thread::Thread::disp(void*) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbutil.so
#5 0x00007fffee540fa7 in thread_fun(void*) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbutil.so
#6 0x00007fffee54421c in sb_thread_sthr_disp(void*) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbutil.so
#7 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
#8 0x0000003e3bce890d in clone () from /lib64/libc.so.6
Thread 29 (Thread 0x7fffed6a6700 (LWP 20112)):
#0 0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fffee5435e2 in SB_Thread::CV::wait() () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbutil.so
#2 0x00007fffee5436be in SB_Thread::CV::wait(bool) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbutil.so
#3 0x00007fffefebac47 in SB_Sig_Queue::remove() () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbms.so
#4 0x00007fffefec8977 in SB_Trans::Sock_Stream_Helper_Thread::run() () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbms.so
#5 0x00007fffefec823a in sock_helper_thread_fun(void*) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbms.so
#6 0x00007fffee540b4f in SB_Thread::Thread::disp(void*) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbutil.so
#7 0x00007fffee540fa7 in thread_fun(void*) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbutil.so
#8 0x00007fffee54421c in sb_thread_sthr_disp(void*) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbutil.so
#9 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
#10 0x0000003e3bce890d in clone () from /lib64/libc.so.6
Thread 28 (Thread 0x7fffead82700 (LWP 20113)):
#0 0x0000003e3bc3399d in sigtimedwait () from /lib64/libc.so.6
#1 0x00007fffefe865c4 in local_monitor_reader(void*) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbms.so
#2 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
#3 0x0000003e3bce890d in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7fffea301700 (LWP 20115)):
#0 0x0000003e3c00ed2d in nanosleep () from /lib64/libpthread.so.0
#1 0x00007fffef6032b9 in Sleep(unsigned int) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libwin.so
#2 0x00007ffff7747d27 in memMonitorUpdateThread(void*) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcli.so
#3 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
#4 0x0000003e3bce890d in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7fffe977e700 (LWP 20116)):
#0 0x0000003e3bce8f03 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fffefec0cd5 in SB_Trans::Sock_Controller::epoll_wait(char const*, int) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbms.so
#2 0x00007fffefec01f3 in SB_Trans::Sock_Comp_Thread::run() () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbms.so
#3 0x00007fffefec002d in sock_comp_thread_fun(void*) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbms.so
#4 0x00007fffee540b4f in SB_Thread::Thread::disp(void*) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbutil.so
#5 0x00007fffee540fa7 in thread_fun(void*) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbutil.so
#6 0x00007fffee54421c in sb_thread_sthr_disp(void*) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbutil.so
#7 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
#8 0x0000003e3bce890d in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7fffe523e700 (LWP 20117)):
#0 0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff17e3243 in os::PlatformEvent::park() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff17aaaaf in Monitor::IWait(Thread*, long) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff17ab23e in Monitor::wait(bool, long, bool) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff151f9e3 in GCTaskManager::get_task(unsigned int) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#5 0x00007ffff15210a8 in GCTaskThread::run() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#6 0x00007ffff17e9988 in java_start(Thread*) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#7 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
#8 0x0000003e3bce890d in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7fffe513d700 (LWP 20118)):
#0 0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff17e3243 in os::PlatformEvent::park() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff17aaaaf in Monitor::IWait(Thread*, long) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff17ab23e in Monitor::wait(bool, long, bool) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff151f9e3 in GCTaskManager::get_task(unsigned int) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#5 0x00007ffff15210a8 in GCTaskThread::run() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#6 0x00007ffff17e9988 in java_start(Thread*) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#7 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
#8 0x0000003e3bce890d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7fffe503c700 (LWP 20119)):
#0 0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff17e3243 in os::PlatformEvent::park() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff17aaaaf in Monitor::IWait(Thread*, long) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff17ab23e in Monitor::wait(bool, long, bool) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff151f9e3 in GCTaskManager::get_task(unsigned int) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#5 0x00007ffff15210a8 in GCTaskThread::run() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#6 0x00007ffff17e9988 in java_start(Thread*) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#7 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
#8 0x0000003e3bce890d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7fffe4f3b700 (LWP 20120)):
#0 0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff17e3243 in os::PlatformEvent::park() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff17aaaaf in Monitor::IWait(Thread*, long) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff17ab23e in Monitor::wait(bool, long, bool) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff151f9e3 in GCTaskManager::get_task(unsigned int) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#5 0x00007ffff15210a8 in GCTaskThread::run() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#6 0x00007ffff17e9988 in java_start(Thread*) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#7 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
#8 0x0000003e3bce890d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7fffe4e3a700 (LWP 20121)):
#0 0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff17e3243 in os::PlatformEvent::park() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff17aaaaf in Monitor::IWait(Thread*, long) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff17ab23e in Monitor::wait(bool, long, bool) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff151f9e3 in GCTaskManager::get_task(unsigned int) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#5 0x00007ffff15210a8 in GCTaskThread::run() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#6 0x00007ffff17e9988 in java_start(Thread*) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#7 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
#8 0x0000003e3bce890d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7fffe4d39700 (LWP 20122)):
#0 0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff17e3243 in os::PlatformEvent::park() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff17aaaaf in Monitor::IWait(Thread*, long) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff17ab23e in Monitor::wait(bool, long, bool) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff151f9e3 in GCTaskManager::get_task(unsigned int) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#5 0x00007ffff15210a8 in GCTaskThread::run() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#6 0x00007ffff17e9988 in java_start(Thread*) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#7 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
#8 0x0000003e3bce890d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7fffe4c38700 (LWP 20123)):
#0 0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff17e3243 in os::PlatformEvent::park() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff17aaaaf in Monitor::IWait(Thread*, long) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff17ab23e in Monitor::wait(bool, long, bool) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff151f9e3 in GCTaskManager::get_task(unsigned int) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#5 0x00007ffff15210a8 in GCTaskThread::run() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#6 0x00007ffff17e9988 in java_start(Thread*) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#7 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
#8 0x0000003e3bce890d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7fffe4b37700 (LWP 20124)):
#0 0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff17e3243 in os::PlatformEvent::park() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff17aaaaf in Monitor::IWait(Thread*, long) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff17ab23e in Monitor::wait(bool, long, bool) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff151f9e3 in GCTaskManager::get_task(unsigned int) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#5 0x00007ffff15210a8 in GCTaskThread::run() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#6 0x00007ffff17e9988 in java_start(Thread*) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#7 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
#8 0x0000003e3bce890d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7fffe4a36700 (LWP 20125)):
#0 0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff17e3243 in os::PlatformEvent::park() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff17aaaaf in Monitor::IWait(Thread*, long) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff17ab23e in Monitor::wait(bool, long, bool) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff151f9e3 in GCTaskManager::get_task(unsigned int) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#5 0x00007ffff15210a8 in GCTaskThread::run() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#6 0x00007ffff17e9988 in java_start(Thread*) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#7 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
#8 0x0000003e3bce890d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7fffe4935700 (LWP 20126)):
#0 0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff17e3243 in os::PlatformEvent::park() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff17aaaaf in Monitor::IWait(Thread*, long) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff17ab23e in Monitor::wait(bool, long, bool) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff151f9e3 in GCTaskManager::get_task(unsigned int) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#5 0x00007ffff15210a8 in GCTaskThread::run() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#6 0x00007ffff17e9988 in java_start(Thread*) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#7 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
#8 0x0000003e3bce890d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7fffdfdec700 (LWP 20127)):
#0 0x0000003e3c00b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff17e88ef in os::PlatformEvent::park(long) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff17aad5e in Monitor::IWait(Thread*, long) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff17ab23e in Monitor::wait(bool, long, bool) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff1970169 in VMThread::loop() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#5 0x00007ffff1970470 in VMThread::run() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#6 0x00007ffff17e9988 in java_start(Thread*) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#7 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
#8 0x0000003e3bce890d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7fffdfceb700 (LWP 20128)):
#0 0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff17e3243 in os::PlatformEvent::park() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff17d7add in ObjectMonitor::wait(long, bool, Thread*) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff1647378 in JVM_MonitorWait () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#4 0x00007fffe56ba7f8 in ?? ()
#5 0x00007fffdfcd3390 in ?? ()
#6 0x0000000000c33ca0 in ?? ()
#7 0x00007fffdfcd31c0 in ?? ()
#8 0x00007fffdfcd3158 in ?? ()
#9 0x0000000000000000 in ?? ()
Thread 13 (Thread 0x7fffdfbea700 (LWP 20129)):
#0 0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff17e3243 in os::PlatformEvent::park() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff17d7add in ObjectMonitor::wait(long, bool, Thread*) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff1647378 in JVM_MonitorWait () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#4 0x00007fffe56ba7f8 in ?? ()
#5 0x00007fffdfbd22e0 in ?? ()
#6 0x0000000000c3e000 in ?? ()
#7 0x0000000000c38210 in ?? ()
#8 0x00007fffdfbd2268 in ?? ()
#9 0x0000000000000000 in ?? ()
Thread 12 (Thread 0x7fffdfae9700 (LWP 20130)):
#0 0x0000003e3c00d720 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007ffff17e85ca in check_pending_signals(bool) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff17e1835 in signal_thread_entry(JavaThread*, Thread*) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff1922d9f in JavaThread::thread_main_inner() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff1922ea5 in JavaThread::run() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#5 0x00007ffff17e9988 in java_start(Thread*) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#6 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003e3bce890d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7fffdf9e8700 (LWP 20131)):
#0 0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff17e3243 in os::PlatformEvent::park() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff17aaaaf in Monitor::IWait(Thread*, long) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff17ab2b6 in Monitor::wait(bool, long, bool) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff1414ad8 in CompileQueue::get() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#5 0x00007ffff1418bda in CompileBroker::compiler_thread_loop() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#6 0x00007ffff1922d9f in JavaThread::thread_main_inner() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#7 0x00007ffff1922ea5 in JavaThread::run() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#8 0x00007ffff17e9988 in java_start(Thread*) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#9 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
#10 0x0000003e3bce890d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7fffdf8e7700 (LWP 20132)):
#0 0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff17e3243 in os::PlatformEvent::park() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff17aaaaf in Monitor::IWait(Thread*, long) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff17ab2b6 in Monitor::wait(bool, long, bool) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff1414ad8 in CompileQueue::get() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#5 0x00007ffff1418bda in CompileBroker::compiler_thread_loop() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#6 0x00007ffff1922d9f in JavaThread::thread_main_inner() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#7 0x00007ffff1922ea5 in JavaThread::run() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#8 0x00007ffff17e9988 in java_start(Thread*) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#9 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
#10 0x0000003e3bce890d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7fffdf7e6700 (LWP 20133)):
#0 0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff17e3243 in os::PlatformEvent::park() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff17aaaaf in Monitor::IWait(Thread*, long) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff17ab23e in Monitor::wait(bool, long, bool) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff1879388 in ServiceThread::service_thread_entry(JavaThread*, Thread*) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#5 0x00007ffff1922d9f in JavaThread::thread_main_inner() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#6 0x00007ffff1922ea5 in JavaThread::run() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#7 0x00007ffff17e9988 in java_start(Thread*) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#8 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
#9 0x0000003e3bce890d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7fffdf6e5700 (LWP 20134)):
#0 0x0000003e3c00b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff17e88ef in os::PlatformEvent::park(long) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff17aad5e in Monitor::IWait(Thread*, long) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff17ab23e in Monitor::wait(bool, long, bool) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff191edc1 in WatcherThread::sleep() const () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#5 0x00007ffff191f2de in WatcherThread::run() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#6 0x00007ffff17e9988 in java_start(Thread*) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#7 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
#8 0x0000003e3bce890d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7fffde317700 (LWP 20137)):
#0 0x0000003e3c00b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff17e8114 in Parker::park(bool, long) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff1946a05 in Unsafe_Park () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#3 0x00007fffe56ba7f8 in ?? ()
#4 0x0000000000bb4968 in ?? ()
#5 0x0000000000000000 in ?? ()
Thread 6 (Thread 0x7fffddedb700 (LWP 20141)):
#0 0x0000003e3bcdf253 in poll () from /lib64/libc.so.6
#1 0x00007fffde32a2ca in Java_org_apache_hadoop_net_unix_DomainSocketWatcher_doPoll0 () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/sql/local_hadoop/hadoop-2.5.0-cdh5.3.0/lib/native/libhadoop.so.1.0.0
#2 0x00007fffe56ba7f8 in ?? ()
#3 0x00000000020a4800 in ?? ()
#4 0x0000000000000000 in ?? ()
Thread 5 (Thread 0x7fffde2de700 (LWP 20153)):
#0 0x0000003e3bce8f03 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fffde96b707 in Java_sun_nio_ch_EPollArrayWrapper_epollWait () from /opt/home/tools/jdk1.7.0_67/jre/lib/amd64/libnio.so
#2 0x00007fffe56ba7f8 in ?? ()
#3 0x0000000000000000 in ?? ()
Thread 4 (Thread 0x7fffddbd8700 (LWP 20154)):
#0 0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff17e8228 in Parker::park(bool, long) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff1946a05 in Unsafe_Park () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#3 0x00007fffe56ba7f8 in ?? ()
#4 0x00007fffe56ae058 in ?? ()
#5 0x0000000000000000 in ?? ()
Thread 3 (Thread 0x7fffddfdc700 (LWP 20156)):
#0 0x0000003e3c00b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff17e8114 in Parker::park(bool, long) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff1946a05 in Unsafe_Park () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#3 0x00007fffe56ba7f8 in ?? ()
#4 0x00007fffe56ae058 in ?? ()
#5 0x0000000000000000 in ?? ()
Thread 2 (Thread 0x7fffdddda700 (LWP 20236)):
#0 0x0000003e3c00b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff17e8114 in Parker::park(bool, long) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff1946a05 in Unsafe_Park () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#3 0x00007fffe56ba7f8 in ?? ()
#4 0x00007fffe56ae058 in ?? ()
#5 0x0000000000000000 in ?? ()
Thread 1 (Thread 0x7fffee315c00 (LWP 20110)):
#0 0x0000003e3c00b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fffee543778 in SB_Thread::CV::wait(int, int) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbutil.so
#2 0x00007fffee54383d in SB_Thread::CV::wait(bool, int, int) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbutil.so
#3 0x00007fffefe988cb in SB_Ms_Event_Mgr::wait(long) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbms.so
#4 0x00007fffefeb916e in XWAIT_com(short, int, bool) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbms.so
#5 0x00007fffefeb8f1f in XWAIT(short, int) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbms.so
#6 0x00007fffefc3731a in fs_int_fs_file_awaitiox(short*, void*, int, long*, int, short*, bool, bool) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbfs.so
#7 0x00007fffefc30e15 in BAWAITIOXTS(short*, void*, int, long*, int, short*) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbfs.so
#8 0x00007ffff70be1f7 in GuaConnectionToServer::wait(int, unsigned int*, IpcAwaitiox*) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcommon.so
#9 0x00007ffff70b51cb in IpcSetOfConnections::waitOnSet(int, int, int*) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcommon.so
#10 0x00007ffff70b633d in IpcMessageStream::waitOnMsgStream(int) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcommon.so
#11 0x00007ffff70b6692 in IpcMessageStream::receive(int) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcommon.so
#12 0x00007ffff70bdf42 in GuaConnectionToServer::tryToStartNewIO() () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcommon.so
#13 0x00007ffff70beb38 in GuaConnectionToServer::send(IpcMessageBuffer*) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcommon.so
#14 0x00007ffff70b72e4 in IpcMessageStream::send(int, long) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcommon.so
#15 0x00007ffff773fad5 in ExSqlComp::sendRequest(CmpMessageObj*, int) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcli.so
#16 0x00007ffff773e620 in ExSqlComp::sendRequest(CmpMessageObj::MessageTypeEnum, char const*, unsigned int, int, long*, int, int, char const*, int) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcli.so
#17 0x00007ffff775fdd9 in CliStatement::prepare2(char*, ComDiagsArea&, char*, unsigned int, int, int, unsigned int) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcli.so
#18 0x00007ffff7760846 in CliStatement::prepare(char*, ComDiagsArea&, char*, unsigned int, int, int, unsigned int) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcli.so
#19 0x00007ffff77130d4 in SQLCLI_Prepare2 () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcli.so
#20 0x00007ffff776dc6e in SQL_EXEC_Prepare2 () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcli.so
#21 0x00007ffff3c819d2 in ExeCliInterface::prepare(char const*, SQLMODULE_ID_t*, SQLCLI_OBJ_ID*, SQLCLI_OBJ_ID*, SQLCLI_OBJ_ID*, SQLCLI_OBJ_ID*, char*, Queue, char*, Queue, char*, int*, SQL_QUERY_COST_INFO*, SQL_QUERY_COMPILER_STATS_INFO*, int) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libexecutor.so
#22 0x00007ffff3c83460 in ExeCliInterface::fetchRowsPrologue(char const*, int, int, char*) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libexecutor.so
#23 0x00007ffff7a627fb in CmpSeabaseDDL::getObjectUIDandOwners(ExeCliInterface*, char const*, char const*, char const*, ComObjectType, int&, int&, bool, int) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsqlcomp.so
#24 0x00007ffff7ae763d in CmpSeabaseDDL::getSeabaseUserTableDesc(NAString const&, NAString const&, NAString const&, ComObjectType, int) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsqlcomp.so
#25 0x00007ffff7aea687 in CmpSeabaseDDL::getSeabaseTableDesc(NAString const&, NAString const&, NAString const&, ComObjectType, int) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsqlcomp.so
#26 0x00007ffff4d5d304 in NATableDB::get(CorrName&, BindWA*, desc_struct*) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/liboptimizer.so
#27 0x00007ffff4ab2efc in BindWA::getNATable(CorrName&, int, desc_struct*) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/liboptimizer.so
#28 0x00007ffff4ad361e in Scan::bindNode(BindWA*) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/liboptimizer.so
#29 0x00007ffff4a9fe07 in RelExpr::bindChildren(BindWA*) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/liboptimizer.so
#30 0x00007ffff4adffde in RelRoot::bindNode(BindWA*) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/liboptimizer.so
#31 0x00007ffff7a3f50e in CmpMain::compile(char const*, int, RelExpr*&, char*, unsigned int, NAMemory*, CmpMain::CompilerPhase, FragmentDir*, int, int, int, timeval*, int) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsqlcomp.so
#32 0x00007ffff7a4232c in CmpMain::sqlcomp(char const*, int, RelExpr*&, char*, unsigned int, NAMemory*, CmpMain::CompilerPhase, FragmentDir*, int, int, int, timeval*, int) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsqlcomp.so
#33 0x00007ffff7a43630 in CmpMain::sqlcomp(QueryText&, int, char*, unsigned int, NAMemory*, CmpMain::CompilerPhase, FragmentDir**, int) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsqlcomp.so
#34 0x00007ffff40dfc08 in CmpStatement::process(CmpMessageSQLText const&) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libarkcmplib.so
#35 0x00007ffff40ca93d in ExCmpMessage::actOnReceive(IpcConnection*) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libarkcmplib.so
#36 0x00007ffff70a58d6 in IpcMessageStream::internalActOnReceive(IpcMessageBuffer*, IpcConnection*) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcommon.so
#37 0x00007ffff70c02fb in GuaConnectionToClient::acceptBuffer(IpcMessageBuffer*, unsigned int) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcommon.so
#38 0x00007ffff70c100f in GuaReceiveControlConnection::wait(int, unsigned int*, IpcAwaitiox*) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcommon.so
#39 0x00007ffff70c23b2 in GuaConnectionToClient::wait(int, unsigned int*, IpcAwaitiox*) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcommon.so
#40 0x00007ffff70b54aa in IpcSetOfConnections::waitOnSet(int, int, int*) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcommon.so
#41 0x00007ffff70b633d in IpcMessageStream::waitOnMsgStream(int) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcommon.so
#42 0x00007ffff70b6692 in IpcMessageStream::receive(int) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcommon.so
#43 0x00000000004064ae in main ()

One more quick note, even after the sqlci session ended,
only the initial tdm_arkcmp process was cleaned up.

The others remained:

Processing cluster.conf on local host g4t3037.houston.hp.com
[$Z000IRF] Shell/shell Version 1.0.1 Release 1.1.0 e 26Mar15)
[$Z000IRF] %ps
[$Z000IRF] NID,PID(os) PRI TYPE STATES NAME PARENT [$Z000IRF] ------------ — ---- ------- ----------- [$Z000IRF] 000,00018562 000 WDG ES-A- $WDG000 NONE [$Z000IRF] 000,00018563 000 PSD ES-A- $PSD000 NONE [$Z000IRF] 000,00018592 001 DTM ES-A- $TM0 [$Z000IRF] 000,00019243 001 GEN ES-A- $ZSC000 NONE [$Z000IRF] 000,00019274 001 SSMP ES-A- $ZSM000 NONE [$Z000IRF] 000,00020982 001 GEN ES-A- $ZLOBSRV0 NONE [$Z000IRF] 000,00007416 001 GEN ES-A- $Z00061W NONE [$Z000IRF] 000,00007960 001 GEN ES-A- $Z0006HF [$Z000IRF] 000,00008021 001 GEN ES-A- $Z0006J6 [$Z000IRF] 000,00008079 001 GEN ES-A- $Z0006KU [$Z000IRF] 000,00008137 001 GEN ES-A- $Z0006MH [$Z000IRF] 000,00008194 001 GEN ES-A- $Z0006P4 [$Z000IRF] 000,00008252 001 GEN ES-A- $Z0006QS [$Z000IRF] 000,00008312 001 GEN ES-A- $Z0006SH [$Z000IRF] 000,00008369 001 GEN ES-A- $Z0006U4 [$Z000IRF] 000,00008427 001 GEN ES-A- $Z0006VS … (Build release [joaniec], dat
PROGRAM
----------- ---------------
sqwatchdog
pstartd
NONE tm
mxsscp
mxssmp
mxlobsrvr
tdm_arkcmp
$Z00061W tdm_arkcmp
$Z0006HF tdm_arkcmp
$Z0006J6 tdm_arkcmp
$Z0006KU tdm_arkcmp
$Z0006MH tdm_arkcmp
$Z0006P4 tdm_arkcmp
$Z0006QS tdm_arkcmp
$Z0006SH tdm_arkcmp
$Z0006U4 tdm_arkcmp

The new memoryUsage threshold and throttle capabilities were used to initially
see the problem.

This was on the CDH 5.3 workstation using the install_local_hadoop support
on Trafidion R1.1.0 build version git032415 or later build.

Attachments

Activity

People

Assignee:: Qifan Chen

Reporter:: Joanie Cooper

Votes:: 0 Vote for this issue

Watchers:: 1 Start watching this issue

Dates

Created:: 22/Jul/15 18:19

Updated:: 06/Mar/18 16:15