[VCL-1073] Linux.pm::reboot may fail because it doesn't detect broken SSH connection during first attempt

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.2
• Fix Version/s: 2.5
• Component/s: vcld (backend)
• Assignee: Andrew Kurth
• Reporter: Andrew Kurth
• Created: 07/Aug/17 17:24
• Updated: 10/Aug/17 15:33
• Resolved: 10/Aug/17 15:33

Description

On occasion, the SSH connection to the machine being rebooted is not gracefully terminated after shutdown -r is executed by Linux.pm::reboot. The SSH command returns Connection to <VM IP address> closed by remote host.:

utils.pm:run_ssh_command|4382| ---- WARNING ----
utils.pm:run_ssh_command|4382| 2017-08-05 14:04:32utils.pm:run_ssh_command|4382|attempt 1/3: failed to execute SSH command on <VM IP address> (<VM>): '/sbin/shutdown -r now', exit status: 255, SSH exits with the exit status of the remote command or with 255 if an error occurred, output 2:
utils.pm:run_ssh_command|4382| ssh output (/sbin/shut...): Connection to <VM IP address> closed by remote host.
utils.pm:run_ssh_command|4382| ( 0) utils.pm, run_ssh_command (line: 4382)
utils.pm:run_ssh_command|4382| (-1) OS.pm, execute (line: 3281)
utils.pm:run_ssh_command|4382| (-2) Linux.pm, reboot (line: 2714)
utils.pm:run_ssh_command|4382| (-3) inuse.pm, process (line: 123)
utils.pm:run_ssh_command|4382| (-4) vcld, make_new_child (line: 601)
utils.pm:run_ssh_command|4382| (-5) vcld, main (line: 353)

utils.pm::run_ssh_command then makes 2 more attempts to call shutdown -r. Both of which expectedly fail because the computer is already in the process of rebooting.

utils.pm:run_ssh_command|4363| ---- WARNING ----
utils.pm:run_ssh_command|4363| attempt 2/3: failed to execute SSH command on <VM IP address> (<VM>): '/sbin/shutdown -r now', exit status: 255, output:
utils.pm:run_ssh_command|4363| ssh output (/sbin/shut...): ssh: connect to host <VM IP address> port 22: Connection refused
...
utils.pm:run_ssh_command|4400| ---- WARNING ----
utils.pm:run_ssh_command|4400| failed to run SSH command after 3 attempts, command: /usr/bin/ssh -i /etc/vcl/vcl.key -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ConnectionAttempts=1 -o ConnectTimeout=30 -o BatchMode=no -o PasswordAuthentication=no -l root -p 22 -x <VM IP address> '/sbin/shutdown -r now' 2>&1, exit status: 255, output:
utils.pm:run_ssh_command|4400| ssh output (/sbin/shut...): ssh: connect to host <VM IP address> port 22: Connection timed out
...
Linux.pm:reboot|2716| ---- WARNING ----
Linux.pm:reboot|2716| failed to execute command to reboot <VM>

If the user initiated a reboot, a CRITICAL message is sent:

inuse.pm:process|123| ---- CRITICAL ----
inuse.pm:process|123| user requested reboot of <VM> failed