[IMPALA-10842] Show full stacktraces of jvm threads in web UI

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Component/s: Frontend
• Labels: observability ramp-up
• Epic Color: ghx-label-10
• Assignee: Unassigned
• Reporter: Quanlong Huang
• Created: 05/Aug/21 03:04
• Updated: 08/May/23 23:58
• Resolved: 08/May/23 23:58

Description

IMPALA-4885 adds a web page showing all the JVM threads, which saves us some time to ssh to the machine and perform jstack manually.

However, the summary column doesn't show the full stacktrace if it has more than 8 frames. For instance, when debugging a long running DDL in catalogd, I search for "execDdl" in the search box and get nothing since "execDdl" is truncated:

I finally did a jstack and find the thread:

"Thread-11" #42 prio=5 os_prio=0 tid=0x000000000f86c000 nid=0x4c99 runnable [0x00007f9a8f8e1000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
        - locked <0x00000005d4dbfa88> (a java.io.BufferedInputStream)
        at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
        at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
        at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:425)
        at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:321)
        at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:225)
        at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table_objects_by_name_req(ThriftHiveMetastore.java:2492)
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table_objects_by_name_req(ThriftHiveMetastore.java:2479)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTables(HiveMetaStoreClient.java:2333)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTableObjectsByName(HiveMetaStoreClient.java:2314)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
        at com.sun.proxy.$Proxy11.getTableObjectsByName(Unknown Source)
        at org.apache.impala.service.CatalogOpExecutor.dropTablesFromKudu(CatalogOpExecutor.java:2349)
        at org.apache.impala.service.CatalogOpExecutor.dropDatabase(CatalogOpExecutor.java:2255)
        - locked <0x00000005d4a101e8> (a java.lang.Object)
        at org.apache.impala.service.CatalogOpExecutor.execDdlRequest(CatalogOpExecutor.java:462)
        at org.apache.impala.service.JniCatalog.execDdl(JniCatalog.java:240)

The cause is that we simply use TheadInfo#toString() which truncates the stacktrace: https://github.com/apache/impala/blob/47d868fe586905b2f4ea347a97ce582a45aac405/fe/src/main/java/org/apache/impala/common/JniUtil.java#L281

We can improve this to get the whole stacktrace.

Issue Links

• duplicates: IMPALA-5392 Stack depth for threads printed in the Catalog UI under JVM Threads is not deep enough