Critical Description
Glog 0.4 introduced supported for using thread local storage for its buffer. This feature is controlled by the WITH_TLS CMake variable, and it defaults to ON. See https://github.com/google/glog/commit/2df0ca34aa3000dadf76633ca700abf0bf50756d . When Kudu upgraded to glog 0.6.0 as part of the M1 fixes in "KUDU-3374 Add support for M1 and macOS Monterey", it increased the thread local storage usage by >30000 bytes.
# Older libkudu_client.so has 0x100 = 256 bytes of TLS: $ readelf -l libkudu_client.so | grep "TLS" -A1 TLS 0x00000000007d14c0 0x00000000007d24c0 0x00000000007d24c0 0x0000000000000080 0x0000000000000100 R 0x40 # Newer libkudu_client.so has 0x77b9 = 30649 bytes of TLS: $ readelf -l libkudu_client.so.0 | grep TLS -A1 TLS 0x0000000000751280 0x0000000000752280 0x0000000000752280 0x0000000000000080 0x00000000000077b9 R 40
This is a problem for Impala, because Impala starts a JVM. There are certain JVM threads (like the "reaper thread") that have very small stacks (e.g. 32KB) and with glibc the TLS space is allocated at the expense of stack space. 30k of TLS usage leaves very little for the reaper thread. There are a series of bugs where the Java reaper thread hits a StackOverflowException because of high TLS usage. This can cause various symptoms including hangs.
GLIBC message thread: https://sourceware.org/bugzilla/show_bug.cgi?id=11787
JDK bugs: JDK-8217475, JDK-8225035
To resolve Impala's problem, it would be useful to build libkudu_client.so with glog's WITH_TLS=OFF.