[IMPALA-12011] Shell command of webserver_private_key_password_cmd doesn't unset JAVA_TOOL_OPTIONS variable

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: Impala 4.3.0
• Component/s: Backend
• Target Version: Impala 4.1.3
• Epic Color: ghx-label-10
• Assignee: Quanlong Huang
• Reporter: Quanlong Huang
• Created: 21/Mar/23 07:22
• Updated: 23/Apr/23 08:55
• Resolved: 23/Apr/23 08:55

Description

I'm using IntelliJ to attach to a catalog server in a secured cluster (Kerberos+SSL), I set JAVA_TOOL_OPTIONS to

-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=30020

However, catalogd fails to start since the java process launched by the webserver_private_key_password_cmd also inherits this env variable. Hence failed to bind the port of 30020 which is already used by catalogd.

ERROR: transport error 202: bind failed: Address already in use
ERROR: JDWP Transport dt_socket failed to initialize, TRANSPORT_INIT(510)
JDWP exit error AGENT_ERROR_TRANSPORT_INIT(197): No transports initialized [debugInit.c:750]
F0321 06:23:42.424225 577816 catalogd-main.cc:70] SSL private-key password command ('/var/run/cloudera-scm-agent/process/63-impala-CATALOGSERVER/altscript.sh sec-0-webserver_private_key_password_cmd') failed with error: Shell cmd: '/var/run/cloudera-scm-agent/process/63-impala-CATALOGSERVER/altscript.sh sec-0-webserver_private_key_password_cmd' exited with error status: '2'. Stdout was: ''
. Impalad exiting.
*** Check failure stack trace: ***
    @          0x2e3d5ec
    @          0x2e3eedc
    @          0x2e3cf4a
    @          0x2e40b48
    @           0xbdfec7
    @           0xb54a07
    @     0x7feabb01d444
    @           0xbdeb96
Picked up JAVA_TOOL_OPTIONS: -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=30020 

IMPALA-6293 is supposed to address this issue. But the patch of it has a mistack in
be/src/util/webserver.cc: https://gerrit.cloudera.org/c/12005/7/be/src/util/webserver.cc

const string& password_cmd = FLAGS_webserver_private_key_password_cmd;
if (!password_cmd.empty()) {
  if (!RunShellProcess(password_cmd, &key_password, true)) {
    return Status(TErrorCode::SSL_PASSWORD_CMD_FAILED, password_cmd, key_password,
        {"JAVA_TOOL_OPTIONS"});  // The last argument should be added to the above RunShellProcess() call
  }

Thus the shell command of webserver_private_key_password_cmd doesn't unset the JAVA_TOOL_OPTIONS env variable.

Issue Links

• relates to: IMPALA-6293 Shell commands run by Impala can fail when using the Java debugger