[IMPALA-12979] Wildcard in CLASSPATH might not work in the RPM package

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 3.4.2
• Fix Version/s: Impala 3.4.2
• Component/s: Infrastructure
• Epic Color: ghx-label-9
• Assignee: Quanlong Huang
• Reporter: Quanlong Huang
• Created: 08/Apr/24 12:32
• Updated: 21/Jun/24 22:10
• Resolved: 21/Jun/24 22:10

Description

I tried deploying the RPM package of Impala-3.4.2 (commit 8e9c5a5) on CentOS 7.9 and found launching catalogd failed by the following error (in catalogd.INFO):

Wrote minidump to /var/log/impala-minidumps/catalogd/5e3c8819-0593-4943-555addbc-665470ad.dmp
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x0000000002baf14c, pid=156082, tid=0x00007fec0dce59c0
#
# JRE version: Java(TM) SE Runtime Environment (8.0_141-b15) (build 1.8.0_141-b15)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.141-b15 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  [catalogd+0x27af14c]  llvm::SCEVAddRecExpr::getNumIterationsInRange(llvm::ConstantRange const&, llvm::ScalarEvolution&) const+0x73c
#
# Core dump written. Default location: /opt/impala/core or core.156082
#
# An error report file with more information is saved as:
# /tmp/hs_err_pid156082.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
# 

There are other logs in catalogd.ERROR

Log file created at: 2024/04/08 04:49:28
Running on machine: ccycloud-1.quanlong.root.comops.site
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0408 04:49:28.979386 158187 logging.cc:146] stderr will be logged to this file.
Wrote minidump to /var/log/impala-minidumps/catalogd/6c3f550c-be96-4a5b-61171aac-0de15155.dmp
could not find method getRootCauseMessage from class (null) with signature (Ljava/lang/Throwable;)Ljava/lang/String;
could not find method getStackTrace from class (null) with signature (Ljava/lang/Throwable;)Ljava/lang/String;
FileSystem: loadFileSystems failed error:
(unable to get root cause for java.lang.NoClassDefFoundError)
(unable to get stack trace for java.lang.NoClassDefFoundError)

Resolving the minidump shows me the following stacktrace:

(gdb) bt
#0  0x0000000002baf14c in ?? ()
#1  0x0000000002baee24 in getJNIEnv ()
#2  0x0000000002bacb71 in hdfsBuilderConnect ()
#3  0x00000000012e6ae2 in impala::JniUtil::InitLibhdfs() ()
#4  0x00000000012e7897 in impala::JniUtil::Init() ()
#5  0x0000000000be9297 in impala::InitCommonRuntime(int, char**, bool, impala::TestInfo::Mode) ()
#6  0x0000000000bb604a in CatalogdMain(int, char**) ()
#7  0x0000000000b33f97 in main ()

It indicates something wrong in initializing the JVM. Here are the env vars:

Environment Variables:
JAVA_HOME=/usr/java/jdk1.8.0_141
CLASSPATH=/opt/impala/conf:/opt/impala/jar/*
PATH=/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/root/bin
LD_LIBRARY_PATH=/opt/impala/lib/:/usr/java/jdk1.8.0_141/jre/lib/amd64/server:/usr/java/jdk1.8.0_141/jre/lib/amd64
SHELL=/bin/bash

We use wildcard "*" in the classpath which seems to be the cause. The issue was resolved after using explicit paths in the classpath. Here are what I changed in bin/impala-env.sh:

#export CLASSPATH="/opt/impala/conf:/opt/impala/jar/*"
CLASSPATH=/opt/impala/conf
for jar in /opt/impala/jar/*.jar; do
  CLASSPATH="$CLASSPATH:$jar"
done
export CLASSPATH