[IMPALA-6073] No response from expr-codegen-test/expr-test

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Cannot Reproduce
• Affects Version/s: Impala 2.9.0
• Component/s: Backend
• Environment: ubuntu 14.04.5 LTS
• Epic Color: ghx-label-4
• Assignee: Philip Martin
• Reporter: Jin Chul Kim
• Created: 18/Oct/17 08:22
• Updated: 13/Mar/20 16:27
• Resolved: 13/Mar/20 16:27

Description

I checked out HEAD of Impala repo. There is no any code modification on my local repo.
After full build, ./build/debug/exprs/expr-test and ./build/debug/exprs/expr-codegen-test seem to be hung. The other gtests in be are working appropriately. Some threads are waiting and the other threads are on sleep.

I guess this issue is similar to the deadlock: https://issues.apache.org/jira/browse/HDFS-11851

Please let me know if you have any workaround.

Here are stack traces on expr-codegen-test using GDB:

jinchulkim@ubuntu:~/workspace/Impala/be$ gdb ./build/debug/exprs/expr-codegen-test
...
(gdb) info thread
  Id   Target Id         Frame
  16   Thread 0x7fffe20ae700 (LWP 23374) "expr-codegen-te" pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
  15   Thread 0x7fffe21af700 (LWP 23373) "expr-codegen-te" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  14   Thread 0x7fffe22b0700 (LWP 23372) "expr-codegen-te" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  13   Thread 0x7fffe23b1700 (LWP 23371) "expr-codegen-te" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  12   Thread 0x7fffe24b2700 (LWP 23370) "expr-codegen-te" sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
  11   Thread 0x7fffe2c95700 (LWP 23369) "expr-codegen-te" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  10   Thread 0x7fffe2d96700 (LWP 23368) "expr-codegen-te" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  9    Thread 0x7fffe2e97700 (LWP 23367) "expr-codegen-te" pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
  8    Thread 0x7fffe915a700 (LWP 23366) "expr-codegen-te" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  7    Thread 0x7fffe925b700 (LWP 23365) "expr-codegen-te" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  6    Thread 0x7fffe935c700 (LWP 23364) "expr-codegen-te" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  5    Thread 0x7fffe945d700 (LWP 23363) "expr-codegen-te" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  4    Thread 0x7fffee536700 (LWP 23362) "expr-codegen-te" 0x00007ffff037bb9d in nanosleep () at ../sysdeps/unix/syscall-template.S:81
  3    Thread 0x7fffeed37700 (LWP 23361) "expr-codegen-te" 0x00007ffff037bb9d in nanosleep () at ../sysdeps/unix/syscall-template.S:81
  2    Thread 0x7fffef538700 (LWP 23356) "expr-codegen-te" 0x00007ffff0067dfd in nanosleep () at ../sysdeps/unix/syscall-template.S:81
* 1    Thread 0x7fffef5408c0 (LWP 23150) "expr-codegen-te" __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
(gdb) where
#0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007ffff0376649 in _L_lock_909 () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00007ffff0376470 in __GI___pthread_mutex_lock (mutex=0x7ffff4080600 <jvmMutex>) at ../nptl/pthread_mutex_lock.c:79
#3  0x00007ffff3e7b666 in mutexLock (m=<optimized out>) at /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/os/posix/mutexes.c:28
#4  0x00007ffff3e73a11 in setTLSExceptionStrings (rootCause=0x0, stackTrace=0x0) at /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.c:581
#5  0x00007ffff3e73393 in printExceptionAndFreeV (env=0x307c1d8, exc=0x301adc0, noPrintFlags=<optimized out>, fmt=0x7ffff3e7bf6e "loadFileSystems", ap=0x7fffffffaab0) at /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/exception.c:183
#6  0x00007ffff3e735ef in printExceptionAndFree (env=<optimized out>, exc=<optimized out>, noPrintFlags=<optimized out>, fmt=<optimized out>) at /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/exception.c:213
#7  0x00007ffff3e74880 in getGlobalJNIEnv () at /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.c:463
#8  getJNIEnv () at /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.c:528
#9  0x00007ffff7845a1c in impala::JniUtil::Init () at /home/jinchulkim/workspace/Impala/be/src/util/jni-util.cc:105
#10 0x00007ffff7e3c221 in impala::InitCommonRuntime (argc=1, argv=0x7fffffffb1e8, init_jvm=true, test_mode=impala::TestInfo::BE_TEST) at /home/jinchulkim/workspace/Impala/be/src/common/init.cc:236
#11 0x00000000008333fc in main (argc=1, argv=0x7fffffffb1e8) at /home/jinchulkim/workspace/Impala/be/src/exprs/expr-codegen-test.cc:361
(gdb) thr 2
[Switching to thread 2 (Thread 0x7fffef538700 (LWP 23356))]
#0  0x00007ffff0067dfd in nanosleep () at ../sysdeps/unix/syscall-template.S:81
81      ../sysdeps/unix/syscall-template.S: No such file or directory.
(gdb) where
#0  0x00007ffff0067dfd in nanosleep () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ffff0067c94 in __sleep (seconds=0) at ../sysdeps/unix/sysv/linux/sleep.c:137
#2  0x00007ffff7e3b304 in LogMaintenanceThread () at /home/jinchulkim/workspace/Impala/be/src/common/init.cc:113
#3  0x00007ffff56e4a8e in boost::detail::function::void_function_invoker0<void (*)(), void>::invoke (function_ptr=...) at /home/jinchulkim/workspace/Impala/toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:112
#4  0x00007ffff7b22f22 in boost::function0<void>::operator() (this=0x7fffef537ca0) at /home/jinchulkim/workspace/Impala/toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:767
#5  0x00007ffff78c0ba3 in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*) (name=..., category=..., functor=..., thread_started=0x7fffffffaa60) at /home/jinchulkim/workspace/Impala/be/src/util/thread.cc:352
#6  0x00007ffff78ca32c in boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> >::operator()<void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list0&, int) (this=0x2fb47c0, f=@0x2fb47b8: 0x7ffff78c0884 <impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*)>, a=...)
    at /home/jinchulkim/workspace/Impala/toolchain/boost-1.57.0-p3/include/boost/bind/bind.hpp:457
#7  0x00007ffff78ca26f in boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> > >::operator()() (this=0x2fb47b8)
    at /home/jinchulkim/workspace/Impala/toolchain/boost-1.57.0-p3/include/boost/bind/bind_template.hpp:20
#8  0x00007ffff78ca232 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> > > >::run()
    (this=0x2fb4600) at /home/jinchulkim/workspace/Impala/toolchain/boost-1.57.0-p3/include/boost/thread/detail/thread.hpp:116
#9  0x000000000090c53a in thread_proxy ()
#10 0x00007ffff0374184 in start_thread (arg=0x7fffef538700) at pthread_create.c:312
#11 0x00007ffff00a137d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
(gdb)

Here are the relevant environment variables:

jinchulkim@ubuntu:~/workspace/Impala/be$ echo $JAVA_HOME
/usr/lib/jvm/java-7-openjdk-amd64

jinchulkim@ubuntu:~/workspace/Impala/be$ echo $BOOST_LIBRARYDIR
/usr/lib/x86_64-linux-gnu

jinchulkim@ubuntu:~/workspace/Impala/be$ echo $LD_LIBRARY_PATH
:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server:/home/jinchulkim/workspace/Impala/toolchain/cdh_components/hadoop-2.6.0-cdh5.14.0-SNAPSHOT//lib/native:/home/jinchulkim/workspace/Impala/toolchain/snappy-1.1.4/lib:/home/jinchulkim/workspace/Impala/../Impala-lzo/build:/home/jinchulkim/workspace/Impala/toolchain/gcc-4.9.2/lib64:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server:/home/jinchulkim/workspace/Impala/toolchain/cdh_components/hadoop-2.6.0-cdh5.14.0-SNAPSHOT//lib/native:/home/jinchulkim/workspace/Impala/toolchain/snappy-1.1.4/lib:/home/jinchulkim/workspace/Impala/../Impala-lzo/build:/home/jinchulkim/workspace/Impala/toolchain/gcc-4.9.2/lib64

Issue Links

• is related to: HDFS-11851 getGlobalJNIEnv() may deadlock if exception is thrown