Uploaded image for project: 'VCL'
  1. VCL
  2. VCL-1073

Linux.pm::reboot may fail because it doesn't detect broken SSH connection during first attempt

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.4.2
    • Fix Version/s: 2.5
    • Component/s: vcld (backend)
    • Labels:
      None

      Description

      On occasion, the SSH connection to the machine being rebooted is not gracefully terminated after shutdown -r is executed by Linux.pm::reboot. The SSH command returns Connection to <VM IP address> closed by remote host.:

      utils.pm:run_ssh_command|4382| ---- WARNING ----
      utils.pm:run_ssh_command|4382| 2017-08-05 14:04:32utils.pm:run_ssh_command|4382|attempt 1/3: failed to execute SSH command on <VM IP address> (<VM>): '/sbin/shutdown -r now', exit status: 255, SSH exits with the exit status of the remote command or with 255 if an error occurred, output 2:
      utils.pm:run_ssh_command|4382| ssh output (/sbin/shut...): Connection to <VM IP address> closed by remote host.
      utils.pm:run_ssh_command|4382| ( 0) utils.pm, run_ssh_command (line: 4382)
      utils.pm:run_ssh_command|4382| (-1) OS.pm, execute (line: 3281)
      utils.pm:run_ssh_command|4382| (-2) Linux.pm, reboot (line: 2714)
      utils.pm:run_ssh_command|4382| (-3) inuse.pm, process (line: 123)
      utils.pm:run_ssh_command|4382| (-4) vcld, make_new_child (line: 601)
      utils.pm:run_ssh_command|4382| (-5) vcld, main (line: 353)

      utils.pm::run_ssh_command then makes 2 more attempts to call shutdown -r. Both of which expectedly fail because the computer is already in the process of rebooting.

      utils.pm:run_ssh_command|4363| ---- WARNING ----
      utils.pm:run_ssh_command|4363| attempt 2/3: failed to execute SSH command on <VM IP address> (<VM>): '/sbin/shutdown -r now', exit status: 255, output:
      utils.pm:run_ssh_command|4363| ssh output (/sbin/shut...): ssh: connect to host <VM IP address> port 22: Connection refused
      ...
      utils.pm:run_ssh_command|4400| ---- WARNING ----
      utils.pm:run_ssh_command|4400| failed to run SSH command after 3 attempts, command: /usr/bin/ssh -i /etc/vcl/vcl.key -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ConnectionAttempts=1 -o ConnectTimeout=30 -o BatchMode=no -o PasswordAuthentication=no -l root -p 22 -x <VM IP address> '/sbin/shutdown -r now' 2>&1, exit status: 255, output:
      utils.pm:run_ssh_command|4400| ssh output (/sbin/shut...): ssh: connect to host <VM IP address> port 22: Connection timed out
      ...
      Linux.pm:reboot|2716| ---- WARNING ----
      Linux.pm:reboot|2716| failed to execute command to reboot <VM>

      If the user initiated a reboot, a CRITICAL message is sent:

      inuse.pm:process|123| ---- CRITICAL ----
      inuse.pm:process|123| user requested reboot of <VM> failed

        Attachments

          Activity

            People

            • Assignee:
              arkurth Andrew Kurth
              Reporter:
              arkurth Andrew Kurth
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: