[IMPALA-5208] Forked breakpad process blocks indefinitely for WaitForContinueSignal and fails new Impalad process at startup

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 2.7.0
• Fix Version/s: Impala 2.9.0
• Component/s: Backend
• Epic Color: ghx-label-4
• Assignee: Lars Volker
• Reporter: Mostafa Mokhtar
• Created: 14/Apr/17 18:37
• Updated: 17/May/17 15:21
• Resolved: 17/May/17 15:21

Description

New Impala process failing to start

E0414 10:17:56.761270 893048 logging.cc:121] stderr will be logged to this file.
E0414 10:17:59.897265 893215 thrift-server.cc:182] ThriftServer 'backend' (on port: 22000) exited due to TException: Could not bind: Transport endpoint is not connected
E0414 10:17:59.897356 893048 thrift-server.cc:171] ThriftServer 'backend' (on port: 22000) did not start correctly
F0414 10:17:59.899677 893048 impalad-main.cc:89] ThriftServer 'backend' (on port: 22000) did not start correctly
. Impalad exiting.

Call stack from hung breakpad fork

(gdb) bt
#0  0x0000000001b80c9f in google_breakpad::ExceptionHandler::WaitForContinueSignal() ()
#1  0x0000000001b80ddd in google_breakpad::ExceptionHandler::ThreadEntry(void*) ()
#2  0x0000000001b805db in google_breakpad::ExceptionHandler::GenerateDump(google_breakpad::ExceptionHandler::CrashContext*) ()
#3  0x0000000000000000 in ?? ()

PS output

 
ps -e --format='pid ppid pgid user args' | grep impala
 383619       1  383612 impala   python2.7 /usr/lib64/cmf/agent/build/env/bin/cmf-redactor /usr/lib64/cmf/service/impala/impala.sh impalad impalad_flags false
 405348  368389  405348 mmokhtar python /opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.14/bin/../lib/impala-shell/impala_shell.py -k
 852304  852233  852304 mmokhtar python /opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.14/bin/../lib/impala-shell/impala_shell.py
 872925       1  383612 impala   /opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.14/lib/impala/sbin-retail/impalad --flagfile=/run/cloudera-scm-agent/process/60723-impala-IMPALAD/impala-conf/impalad_flags
 880656  852233  880656 mmokhtar python /opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.14/bin/../lib/impala-shell/impala_shell.py -k
 881074  852233  881074 mmokhtar python /opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.14/bin/../lib/impala-shell/impala_shell.py -k -i va1335.halxg.cloudera.com
 883949  874541  883948 mmokhtar grep --color=auto impala

Issue Links

• links to: Breakpad Bug #728