Details
-
Bug
-
Status: Open
-
Critical
-
Resolution: Unresolved
-
2.0.0
-
None
-
We hit this on OpenStack, but IMO it's not related to the provider used.
Description
Connecting with SSHJ SshClient sometimes fails with "Exhausted available authentication methods" message.
It can be caused by the issue#153 reported on sshj GitHub.
In our tests we run following scenario on Openstack:
1. start a Node from an Image
2. make some checks (e.g. ports are open)
3. suspend() the Node and wait till its suspended
4. resume() the Node and wait till its ready (running and ports are open again)
5. get SSH client by calling
SshClient sshClient = getComputeServiceContext().utils().sshForNode().apply(nodeMetadata); sshClient.connect();
Sometimes it fails to connect the SSH with exception:
08:27:48.255 [main] ERROR jclouds.ssh - << (cloud-user:rsa[fingerprint(11:50:0c:5f:c9:a4:bc:c0:85:1d:08:d9:9c:d5:b9:52),sha1(ff:aa:5a:8c:80:7f:04:b7:6c:90:b5:dd:91:c4:8e:ea:4f:c3:e7:9f)]@10.8.50.241:22) error acquiring {hostAndPort=10.8.50.241:22, loginUser=cloud-user, ssh=null, connectTimeout=60000, sessionTimeout=60000} (out of retries - max 1): Exhausted available authentication methods
net.schmizz.sshj.userauth.UserAuthException: Exhausted available authentication methods
at net.schmizz.sshj.SSHClient.auth(SSHClient.java:217) ~[sshj-0.12.0.jar:na]
at net.schmizz.sshj.SSHClient.authPublickey(SSHClient.java:316) ~[sshj-0.12.0.jar:na]
at net.schmizz.sshj.SSHClient.authPublickey(SSHClient.java:335) ~[sshj-0.12.0.jar:na]
at org.jclouds.sshj.SSHClientConnection.create(SSHClientConnection.java:163) ~[jclouds-sshj-2.0.0-eapqe.4.jar:2.0.0-eapqe.4]
at org.jclouds.sshj.SSHClientConnection.create(SSHClientConnection.java:49) ~[jclouds-sshj-2.0.0-eapqe.4.jar:2.0.0-eapqe.4]
at org.jclouds.sshj.SshjSshClient.acquire(SshjSshClient.java:195) [jclouds-sshj-2.0.0-eapqe.4.jar:2.0.0-eapqe.4]
at org.jclouds.sshj.SshjSshClient.connect(SshjSshClient.java:225) [jclouds-sshj-2.0.0-eapqe.4.jar:2.0.0-eapqe.4]
at ...
My guess is about 5% of the scenario runs is failing.
We have also a retry-if-fails logic in our code to prevent situations when the OpenSSH server in the Node is not yet running. So it tries to get the SSH client 12-times with 5 seconds sleeps following every unsuccessful connect. These retries don't help to solve this issue.
We've tried to update SSHJ to latest released version 0.15.0, but we still hit this issue.
When we used JSCH implementation (module JschSshClientModule) instead of the SSHJ (module SshjSshClientModule) we didn't see this problem.