Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-5208

Forked breakpad process blocks indefinitely for WaitForContinueSignal and fails new Impalad process at startup

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • Impala 2.7.0
    • Impala 2.9.0
    • Backend
    • None
    • ghx-label-4

    Description

      New Impala process failing to start

      E0414 10:17:56.761270 893048 logging.cc:121] stderr will be logged to this file.
      E0414 10:17:59.897265 893215 thrift-server.cc:182] ThriftServer 'backend' (on port: 22000) exited due to TException: Could not bind: Transport endpoint is not connected
      E0414 10:17:59.897356 893048 thrift-server.cc:171] ThriftServer 'backend' (on port: 22000) did not start correctly
      F0414 10:17:59.899677 893048 impalad-main.cc:89] ThriftServer 'backend' (on port: 22000) did not start correctly
      . Impalad exiting.
      

      Call stack from hung breakpad fork

      (gdb) bt
      #0  0x0000000001b80c9f in google_breakpad::ExceptionHandler::WaitForContinueSignal() ()
      #1  0x0000000001b80ddd in google_breakpad::ExceptionHandler::ThreadEntry(void*) ()
      #2  0x0000000001b805db in google_breakpad::ExceptionHandler::GenerateDump(google_breakpad::ExceptionHandler::CrashContext*) ()
      #3  0x0000000000000000 in ?? ()
      

      PS output

       
      ps -e --format='pid ppid pgid user args' | grep impala
       383619       1  383612 impala   python2.7 /usr/lib64/cmf/agent/build/env/bin/cmf-redactor /usr/lib64/cmf/service/impala/impala.sh impalad impalad_flags false
       405348  368389  405348 mmokhtar python /opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.14/bin/../lib/impala-shell/impala_shell.py -k
       852304  852233  852304 mmokhtar python /opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.14/bin/../lib/impala-shell/impala_shell.py
       872925       1  383612 impala   /opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.14/lib/impala/sbin-retail/impalad --flagfile=/run/cloudera-scm-agent/process/60723-impala-IMPALAD/impala-conf/impalad_flags
       880656  852233  880656 mmokhtar python /opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.14/bin/../lib/impala-shell/impala_shell.py -k
       881074  852233  881074 mmokhtar python /opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.14/bin/../lib/impala-shell/impala_shell.py -k -i va1335.halxg.cloudera.com
       883949  874541  883948 mmokhtar grep --color=auto impala
      

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            lv Lars Volker
            mmokhtar Mostafa Mokhtar
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment