Whirr
  1. Whirr
  2. WHIRR-378

Auth fail when creating a cluster from an EC2 instance

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Invalid
    • Affects Version/s: 0.6.0
    • Fix Version/s: 0.8.0
    • Component/s: service/hadoop
    • Labels:
      None

      Description

      There is a ssh auth problem when creating a hadoop cluster from an EC2 ubuntu instance.

      I've been using the same configuration file from an EC2 computer an a physical one, everything works fine in the physical one, but I keep getting this error in EC2:

      Running configuration script on nodes: [us-east-1/i-c7fde5a6, us-east-1/i-c9fde5a8, us-east-1/i-cbfde5aa]
      <<authenticated>> woke to: net.schmizz.sshj.userauth.UserAuthException: publickey auth failed
      <<authenticated>> woke to: net.schmizz.sshj.userauth.UserAuthException: publickey auth failed

      The user in the virtual machine is new and with valid .ssh keys.

      The hadoop config file is (omitting commented lines):

      whirr.cluster-name=hadoop
      whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,3 hadoop-datanode+hadoop-tasktracker
      whirr.provider=aws-ec2
      whirr.identity=****
      whirr.credential=****
      whirr.hardware-id=c1.xlarge
      whirr.image-id=us-east-1/ami-da0cf8b3
      whirr.location-id=us-east-1

      1. run-test.sh
        0.6 kB
        Adrian Cole
      2. setup-whirr.sh
        0.9 kB
        Adrian Cole

        Activity

        Hide
        Joris Poort added a comment -

        I've been having the same issue - did you come up with a temporary workaround for this?

        Cheers - Joris

        Show
        Joris Poort added a comment - I've been having the same issue - did you come up with a temporary workaround for this? Cheers - Joris
        Hide
        Adrian Cole added a comment -

        do you know what the login user is for this image? I think jclouds default is to try "ubuntu". If not, you'd want to specify whirr.login-user

        http://whirr.apache.org/docs/0.6.0/configuration-guide.html

        Show
        Adrian Cole added a comment - do you know what the login user is for this image? I think jclouds default is to try "ubuntu". If not, you'd want to specify whirr.login-user http://whirr.apache.org/docs/0.6.0/configuration-guide.html
        Hide
        Joris Poort added a comment -

        Thanks Adrian - tried this out, but unfortunately didn't make a difference for me.

        I've had the same issue come up with a configuration that previously worked. I thought that it may be something on the Amazon side, but haven't figured it out.

        Show
        Joris Poort added a comment - Thanks Adrian - tried this out, but unfortunately didn't make a difference for me. I've had the same issue come up with a configuration that previously worked. I thought that it may be something on the Amazon side, but haven't figured it out.
        Hide
        Adrian Cole added a comment -

        huh. ok, well let's leave this issue in. I should have a chance to test this w/debugger soon.

        Show
        Adrian Cole added a comment - huh. ok, well let's leave this issue in. I should have a chance to test this w/debugger soon.
        Hide
        Joris Poort added a comment -

        Ok, great thanks so much. Let me know if there's anything else you'd like me to test. I can recreate the issue repeatedly right now on my configuration. (very similar to original poster - EC2, Ubuntu, CDH)

        Show
        Joris Poort added a comment - Ok, great thanks so much. Let me know if there's anything else you'd like me to test. I can recreate the issue repeatedly right now on my configuration. (very similar to original poster - EC2, Ubuntu, CDH)
        Hide
        Joris Poort added a comment -

        Just tried this on rackspace, same issues. So it seems to be independent of the provider.

        Show
        Joris Poort added a comment - Just tried this on rackspace, same issues. So it seems to be independent of the provider.
        Hide
        Paul Baclace added a comment -

        I see this issue too (in 0.6.0), as far as I can tell from the description, but the upshot is some nodes are deleted as dead on arrival and more nodes are allocated so the cluster is successfully created. BUT I am charged for 1 hour of time on each apparent DOA node.

        One run and found that 2 out of 5 nodes were seemingly dead on arrival (I have many examples from the same day) That is a high failure rate, so I wonder whether it was a false positive DOA. A summary of the trimmed whirr.log below (last 3 digits of i-number):

        1. starting 3 instances/nodes (fbe, fc0, fc2) at 3:37:19
        2. problem with a node (fc2) at 3:38:46 or 87 sec. after node start
        3. starting a new instance/node (01c) at 3:40:14
        4. problem with a another node (01c) at 3:41:19, or 65sec after node start
        5. start a new instance/node (040) at 3:41:22
        6. delete nodes (01c, fc2) at 3:44:34

        The most caused-by ssh error is "net.schmizz.sshj.userauth.UserAuthException: publickey auth failed".

        It looks like the overall error "problem applying options to node" is occurring 10 seconds after opening the socket, so that node is alive to some extent and it does not appear to be an ssh timeout. That this happens about 1 minute after instance start makes me think there could be an implicit timer awaiting boot-up. (These instances are all using the same private ami from instance-store and no EBS volumes.)

        The failed nodes appear to be deleted after sufficient nodes are started up, not when they are determined to be failed. Looking at billing records, I noticed that I am being charged for these failed nodes, so I think this is an important bug to fix.

        ----whirr.log excerpt------
        03:37:19,043 DEBUG [jclouds.compute] << started instances([region=us-west-1, name=i-f9914fbe])
        03:37:19,133 DEBUG [jclouds.compute] << present instances([region=us-west-1, name=i-f9914fbe])
        03:37:19,332 DEBUG [jclouds.compute] << started instances([region=us-west-1, name=i-87914fc0],[region=us-west-1, name=i-85914fc2])
        03:37:19,495 DEBUG [jclouds.compute] << present instances([region=us-west-1, name=i-87914fc0],[region=us-west-1, name=i-85914fc2])

        03:38:46,153 ERROR [jclouds.compute] << problem applying options to node(us-west-1/i-85914fc2)

        03:40:14,460 DEBUG [jclouds.compute] << started instances([region=us-west-1, name=i-5b8e501c])
        03:40:14,547 DEBUG [jclouds.compute] << present instances([region=us-west-1, name=i-5b8e501c])

        03:41:19,691 ERROR [jclouds.compute] << problem applying options to node(us-west-1/i-5b8e501c)

        03:41:22,738 DEBUG [jclouds.compute] << started instances([region=us-west-1, name=i-078e5040])
        03:41:22,831 DEBUG [jclouds.compute] << present instances([region=us-west-1, name=i-078e5040])
        03:44:34,257 INFO [org.apache.whirr.actions.BootstrapClusterAction] Deleting failed node node us-west-1/i-5b8e501c
        03:44:34,259 INFO [org.apache.whirr.actions.BootstrapClusterAction] Deleting failed node node us-west-1/i-85914fc2
        03:46:27,948 INFO [org.apache.whirr.service.FileClusterStateStore] (main) Wrote instances file instances

        The instances file ends up containing: i-f9914fbe i-87914fc0 i-078e5040
        And not containing: i-5b8e501c i-85914fc2

        Show
        Paul Baclace added a comment - I see this issue too (in 0.6.0), as far as I can tell from the description, but the upshot is some nodes are deleted as dead on arrival and more nodes are allocated so the cluster is successfully created. BUT I am charged for 1 hour of time on each apparent DOA node. One run and found that 2 out of 5 nodes were seemingly dead on arrival (I have many examples from the same day) That is a high failure rate, so I wonder whether it was a false positive DOA. A summary of the trimmed whirr.log below (last 3 digits of i-number): 1. starting 3 instances/nodes (fbe, fc0, fc2) at 3:37:19 2. problem with a node (fc2) at 3:38:46 or 87 sec. after node start 3. starting a new instance/node (01c) at 3:40:14 4. problem with a another node (01c) at 3:41:19, or 65sec after node start 5. start a new instance/node (040) at 3:41:22 6. delete nodes (01c, fc2) at 3:44:34 The most caused-by ssh error is "net.schmizz.sshj.userauth.UserAuthException: publickey auth failed". It looks like the overall error "problem applying options to node" is occurring 10 seconds after opening the socket, so that node is alive to some extent and it does not appear to be an ssh timeout. That this happens about 1 minute after instance start makes me think there could be an implicit timer awaiting boot-up. (These instances are all using the same private ami from instance-store and no EBS volumes.) The failed nodes appear to be deleted after sufficient nodes are started up, not when they are determined to be failed. Looking at billing records, I noticed that I am being charged for these failed nodes, so I think this is an important bug to fix. ---- whirr.log excerpt ------ 03:37:19,043 DEBUG [jclouds.compute] << started instances( [region=us-west-1, name=i-f9914fbe] ) 03:37:19,133 DEBUG [jclouds.compute] << present instances( [region=us-west-1, name=i-f9914fbe] ) 03:37:19,332 DEBUG [jclouds.compute] << started instances( [region=us-west-1, name=i-87914fc0] , [region=us-west-1, name=i-85914fc2] ) 03:37:19,495 DEBUG [jclouds.compute] << present instances( [region=us-west-1, name=i-87914fc0] , [region=us-west-1, name=i-85914fc2] ) 03:38:46,153 ERROR [jclouds.compute] << problem applying options to node(us-west-1/i-85914fc2) 03:40:14,460 DEBUG [jclouds.compute] << started instances( [region=us-west-1, name=i-5b8e501c] ) 03:40:14,547 DEBUG [jclouds.compute] << present instances( [region=us-west-1, name=i-5b8e501c] ) 03:41:19,691 ERROR [jclouds.compute] << problem applying options to node(us-west-1/i-5b8e501c) 03:41:22,738 DEBUG [jclouds.compute] << started instances( [region=us-west-1, name=i-078e5040] ) 03:41:22,831 DEBUG [jclouds.compute] << present instances( [region=us-west-1, name=i-078e5040] ) 03:44:34,257 INFO [org.apache.whirr.actions.BootstrapClusterAction] Deleting failed node node us-west-1/i-5b8e501c 03:44:34,259 INFO [org.apache.whirr.actions.BootstrapClusterAction] Deleting failed node node us-west-1/i-85914fc2 03:46:27,948 INFO [org.apache.whirr.service.FileClusterStateStore] (main) Wrote instances file instances The instances file ends up containing: i-f9914fbe i-87914fc0 i-078e5040 And not containing: i-5b8e501c i-85914fc2
        Hide
        Paul Baclace added a comment -

        Here is the stack trace associated with the "problem applying options" error:

        2011-09-17 03:38:37,260 DEBUG [jclouds.compute] (user thread 2) << socket [address=204.236.165.200, port=22] opened
        2011-09-17 03:38:46,153 ERROR [jclouds.compute] (user thread 2) << problem applying options to node(us-west-1/i-85914fc2):
        org.jclouds.rest.AuthorizationException: (ubuntu@204.236.165.200:22) (ubuntu@204.236.165.200:22) error acquiring SSHClient(ubuntu@204.236.165.200:22)
        : Exhausted available authentication methods
        at org.jclouds.sshj.SshjSshClient.propagate(SshjSshClient.java:387)
        at org.jclouds.sshj.SshjSshClient.acquire(SshjSshClient.java:214)
        at org.jclouds.sshj.SshjSshClient.connect(SshjSshClient.java:229)
        at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSsh.call(RunScriptOnNodeAsInitScriptUsingSsh.java:107)
        at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMa
        p.java:150)
        at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMa
        p.java:57)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
        Caused by: net.schmizz.sshj.userauth.UserAuthException: Exhausted available authentication methods
        at net.schmizz.sshj.userauth.UserAuthImpl.authenticate(UserAuthImpl.java:114)
        at net.schmizz.sshj.SSHClient.auth(SSHClient.java:204)
        at net.schmizz.sshj.SSHClient.authPublickey(SSHClient.java:304)
        at net.schmizz.sshj.SSHClient.authPublickey(SSHClient.java:323)
        at org.jclouds.sshj.SshjSshClient$1.create(SshjSshClient.java:183)
        at org.jclouds.sshj.SshjSshClient$1.create(SshjSshClient.java:155)
        at org.jclouds.sshj.SshjSshClient.acquire(SshjSshClient.java:204)
        ... 9 more
        Caused by: net.schmizz.sshj.userauth.UserAuthException: publickey auth failed at net.schmizz.sshj.userauth.UserAuthImpl.handle(UserAuthImpl.java:157)
        at net.schmizz.sshj.transport.TransportImpl.handle(TransportImpl.java:474) at net.schmizz.sshj.transport.Decoder.decode(Decoder.java:127) at net.schmizz.sshj.transport.Decoder.received(Decoder.java:195)
        at net.schmizz.sshj.transport.Reader.run(Reader.java:72)

        Show
        Paul Baclace added a comment - Here is the stack trace associated with the "problem applying options" error: 2011-09-17 03:38:37,260 DEBUG [jclouds.compute] (user thread 2) << socket [address=204.236.165.200, port=22] opened 2011-09-17 03:38:46,153 ERROR [jclouds.compute] (user thread 2) << problem applying options to node(us-west-1/i-85914fc2): org.jclouds.rest.AuthorizationException: (ubuntu@204.236.165.200:22) (ubuntu@204.236.165.200:22) error acquiring SSHClient(ubuntu@204.236.165.200:22) : Exhausted available authentication methods at org.jclouds.sshj.SshjSshClient.propagate(SshjSshClient.java:387) at org.jclouds.sshj.SshjSshClient.acquire(SshjSshClient.java:214) at org.jclouds.sshj.SshjSshClient.connect(SshjSshClient.java:229) at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSsh.call(RunScriptOnNodeAsInitScriptUsingSsh.java:107) at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMa p.java:150) at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMa p.java:57) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: net.schmizz.sshj.userauth.UserAuthException: Exhausted available authentication methods at net.schmizz.sshj.userauth.UserAuthImpl.authenticate(UserAuthImpl.java:114) at net.schmizz.sshj.SSHClient.auth(SSHClient.java:204) at net.schmizz.sshj.SSHClient.authPublickey(SSHClient.java:304) at net.schmizz.sshj.SSHClient.authPublickey(SSHClient.java:323) at org.jclouds.sshj.SshjSshClient$1.create(SshjSshClient.java:183) at org.jclouds.sshj.SshjSshClient$1.create(SshjSshClient.java:155) at org.jclouds.sshj.SshjSshClient.acquire(SshjSshClient.java:204) ... 9 more Caused by: net.schmizz.sshj.userauth.UserAuthException: publickey auth failed at net.schmizz.sshj.userauth.UserAuthImpl.handle(UserAuthImpl.java:157) at net.schmizz.sshj.transport.TransportImpl.handle(TransportImpl.java:474) at net.schmizz.sshj.transport.Decoder.decode(Decoder.java:127) at net.schmizz.sshj.transport.Decoder.received(Decoder.java:195) at net.schmizz.sshj.transport.Reader.run(Reader.java:72)
        Hide
        Andrei Savu added a comment -

        Do you think this is a timing issue? Maybe the keys are not installed on the remote machine when jclouds tries to connect using SSH.

        Show
        Andrei Savu added a comment - Do you think this is a timing issue? Maybe the keys are not installed on the remote machine when jclouds tries to connect using SSH.
        Hide
        Paul Baclace added a comment -

        This is the age of "eventually consistent" which means the app level must deal with inconsistency... but "app" in this case could be jclouds or Whirr, I don't know yet.

        This might be a pass-thru issue to jclouds since the timeouts and number of attempts is not settable [sic] in jclouds.

        (I have also seen an error about a security group not existing but it was created moments earlier: this one I will post as a separate bug since the problem above can occur independently.)

        Show
        Paul Baclace added a comment - This is the age of "eventually consistent" which means the app level must deal with inconsistency... but "app" in this case could be jclouds or Whirr, I don't know yet. This might be a pass-thru issue to jclouds since the timeouts and number of attempts is not settable [sic] in jclouds. (I have also seen an error about a security group not existing but it was created moments earlier: this one I will post as a separate bug since the problem above can occur independently.)
        Hide
        Andrei Savu added a comment -

        You are right. We need to implement automatic retry behavior in many places.

        Show
        Andrei Savu added a comment - You are right. We need to implement automatic retry behavior in many places.
        Hide
        Adrian Cole added a comment - - edited

        I just ran this on 0.7.0, and received a different error ( a transient one related to ec2 ssh keys [1] ).

        However, things operated did come up without the ssh error you mentioned. How are your ssh keys setup? Do you have a ~/.ssh/id_rsa and ~/.ssh/id_rsa.pub file in correct form? Are you destroying your cluster through whirr or elsewhere?

        I'll also try with 0.6.0 now

        – snip –
        Running configuration script on nodes: [us-east-1/i-5032a530, us-east-1/i-5232a532, us-east-1/i-5432a534]
        Configuration script run completed
        Running configuration script on nodes: [us-east-1/i-4c32a52c]
        Configuration script run completed
        Completed configuration of hadoop
        Namenode web UI available at http://ec2-184-72-85-39.compute-1.amazonaws.com:50070
        Jobtracker web UI available at http://ec2-184-72-85-39.compute-1.amazonaws.com:50030
        Wrote Hadoop site file /Users/adriancole/.whirr/hadoop/hadoop-site.xml
        Wrote Hadoop proxy script /Users/adriancole/.whirr/hadoop/hadoop-proxy.sh
        Wrote instances file /Users/adriancole/.whirr/hadoop/instances
        Started cluster of 4 instances

        --snip –
        [1] http://code.google.com/p/jclouds/issues/detail?id=696

        Show
        Adrian Cole added a comment - - edited I just ran this on 0.7.0, and received a different error ( a transient one related to ec2 ssh keys [1] ). However, things operated did come up without the ssh error you mentioned. How are your ssh keys setup? Do you have a ~/.ssh/id_rsa and ~/.ssh/id_rsa.pub file in correct form? Are you destroying your cluster through whirr or elsewhere? I'll also try with 0.6.0 now – snip – Running configuration script on nodes: [us-east-1/i-5032a530, us-east-1/i-5232a532, us-east-1/i-5432a534] Configuration script run completed Running configuration script on nodes: [us-east-1/i-4c32a52c] Configuration script run completed Completed configuration of hadoop Namenode web UI available at http://ec2-184-72-85-39.compute-1.amazonaws.com:50070 Jobtracker web UI available at http://ec2-184-72-85-39.compute-1.amazonaws.com:50030 Wrote Hadoop site file /Users/adriancole/.whirr/hadoop/hadoop-site.xml Wrote Hadoop proxy script /Users/adriancole/.whirr/hadoop/hadoop-proxy.sh Wrote instances file /Users/adriancole/.whirr/hadoop/instances Started cluster of 4 instances --snip – [1] http://code.google.com/p/jclouds/issues/detail?id=696
        Hide
        Adrian Cole added a comment -

        run fine on 0.6.0 with the same properties as are in the issue. I suspect this is actually a content-related problem with your ssh keypair.

        can you try creating a new key and using that instead? this is detailed here:

        http://whirr.apache.org/docs/0.6.0/whirr-in-5-minutes.html

        – snip –
        Running configuration script on nodes: [us-east-1/i-aa21b6ca, us-east-1/i-ac21b6cc, us-east-1/i-ae21b6ce]
        Configuration script run completed
        Running configuration script on nodes: [us-east-1/i-a421b6c4]
        Configuration script run completed
        Completed configuration of hadoop
        Namenode web UI available at http://ec2-50-16-93-2.compute-1.amazonaws.com:50070
        Jobtracker web UI available at http://ec2-50-16-93-2.compute-1.amazonaws.com:50030
        Wrote Hadoop site file /Users/adriancole/.whirr/hadoop/hadoop-site.xml
        Wrote Hadoop proxy script /Users/adriancole/.whirr/hadoop/hadoop-proxy.sh
        Wrote instances file /Users/adriancole/.whirr/hadoop/instances
        Started cluster of 4 instances
        – snip –

        Show
        Adrian Cole added a comment - run fine on 0.6.0 with the same properties as are in the issue. I suspect this is actually a content-related problem with your ssh keypair. can you try creating a new key and using that instead? this is detailed here: http://whirr.apache.org/docs/0.6.0/whirr-in-5-minutes.html – snip – Running configuration script on nodes: [us-east-1/i-aa21b6ca, us-east-1/i-ac21b6cc, us-east-1/i-ae21b6ce] Configuration script run completed Running configuration script on nodes: [us-east-1/i-a421b6c4] Configuration script run completed Completed configuration of hadoop Namenode web UI available at http://ec2-50-16-93-2.compute-1.amazonaws.com:50070 Jobtracker web UI available at http://ec2-50-16-93-2.compute-1.amazonaws.com:50030 Wrote Hadoop site file /Users/adriancole/.whirr/hadoop/hadoop-site.xml Wrote Hadoop proxy script /Users/adriancole/.whirr/hadoop/hadoop-proxy.sh Wrote instances file /Users/adriancole/.whirr/hadoop/instances Started cluster of 4 instances – snip –
        Hide
        Adrian Cole added a comment -

        so, my machine I am testing on is os/x. I will try this from a cloud node running ubuntu in a minute.

        Show
        Adrian Cole added a comment - so, my machine I am testing on is os/x. I will try this from a cloud node running ubuntu in a minute.
        Hide
        Adrian Cole added a comment -

        ok. setting up whirr on slicehost ubuntu natty per below [1]

        note I had to change the referenced setup-whirr.sh to not use the incubating svn repo

        java -jar target/compute-basics-jar-with-dependencies.jar slicehost APIKEY APIKEY mygroup add
        java -jar target/compute-basics-jar-with-dependencies.jar slicehost APIKEY APIKEY mygroup run ~/Downloads/setup-whirr.sh

        [1] http://mail-archives.apache.org/mod_mbox/whirr-dev/201108.mbox/%3CCAARMrHXi1ad5fkGvSYJRYJB7OsDmsOtzLgbR=0G9KNP-FdjBeA@mail.gmail.com%3E

        NEXT STEP:

        make a properties file and run whirr on this fresh install to see if there's an OS issue, or perhaps a setup one.

        Show
        Adrian Cole added a comment - ok. setting up whirr on slicehost ubuntu natty per below [1] note I had to change the referenced setup-whirr.sh to not use the incubating svn repo java -jar target/compute-basics-jar-with-dependencies.jar slicehost APIKEY APIKEY mygroup add java -jar target/compute-basics-jar-with-dependencies.jar slicehost APIKEY APIKEY mygroup run ~/Downloads/setup-whirr.sh [1] http://mail-archives.apache.org/mod_mbox/whirr-dev/201108.mbox/%3CCAARMrHXi1ad5fkGvSYJRYJB7OsDmsOtzLgbR=0G9KNP-FdjBeA@mail.gmail.com%3E NEXT STEP: make a properties file and run whirr on this fresh install to see if there's an OS issue, or perhaps a setup one.
        Hide
        Adrian Cole added a comment -

        works fine. I'll update the issue with the commands I used to create the cluster.

        should try the following in case your keys aren't working right:

        ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa

        Show
        Adrian Cole added a comment - works fine. I'll update the issue with the commands I used to create the cluster. should try the following in case your keys aren't working right: ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
        Hide
        Adrian Cole added a comment -

        scripts ran to setup and run the cluster with the parameters in this issue

        Show
        Adrian Cole added a comment - scripts ran to setup and run the cluster with the parameters in this issue
        Hide
        Adrian Cole added a comment -

        OK I think this testing is done. We need more info before taking any more action, as this config is pretty expensive to test!

        My guess based on the error, is that there is an issue in ~/.ssh/id_rsa or ~/.ssh/id_rsa.pub, something we can possibly validate prior to use. My suggestion is that the user generate new keys and try having whirr use them, especially as this problem is noted in both ec2 and rackspace.

        Show
        Adrian Cole added a comment - OK I think this testing is done. We need more info before taking any more action, as this config is pretty expensive to test! My guess based on the error, is that there is an issue in ~/.ssh/id_rsa or ~/.ssh/id_rsa.pub, something we can possibly validate prior to use. My suggestion is that the user generate new keys and try having whirr use them, especially as this problem is noted in both ec2 and rackspace.
        Hide
        Chris Custine added a comment -

        I have run into this issue as well and have some quick observations. My problems all arose from using whirr.private-key-file and whirr.public-key-file with relative paths like "./key.priv" which seems to work fine with a single instance in the cluster. However when there is more than one instance, the relative paths seem to make some instances fail (but not all). If I add fully qualified paths to the keys, everything works just fine for all instances.

        Highly unscientific, but I did not have time to debug further once I figured it out. Seems to be a bug somewhere, but I am not sure where. If I have time again I will look into it.

        Show
        Chris Custine added a comment - I have run into this issue as well and have some quick observations. My problems all arose from using whirr.private-key-file and whirr.public-key-file with relative paths like "./key.priv" which seems to work fine with a single instance in the cluster. However when there is more than one instance, the relative paths seem to make some instances fail (but not all). If I add fully qualified paths to the keys, everything works just fine for all instances. Highly unscientific, but I did not have time to debug further once I figured it out. Seems to be a bug somewhere, but I am not sure where. If I have time again I will look into it.
        Hide
        Andrei Savu added a comment -

        Is that behaviour happening also on a different machine? Can you tell us more about you setup? Thanks!

        Show
        Andrei Savu added a comment - Is that behaviour happening also on a different machine? Can you tell us more about you setup? Thanks!
        Hide
        Carlos González-Cadenas added a comment -

        Hello,

        I'm running your setup and run scripts attached but I cannot successfully launch a cluster on EC2.

        First I launch in US-EAST-1 a Natty image (ami-fd589594). Then I run the setup script and then the run-test script.

        After creating the instances, whirr starts giving errors and retrying with no success (see below). Can you help us with this?

        Thanks
        Carlos

        Running configuration script on nodes: [us-east-1/i-c70415a4, us-east-1/i-c50415a6, us-east-1/i-cb0415a8]
        <<authenticated>> woke to: net.schmizz.sshj.userauth.UserAuthException: publickey auth failed
        <<authenticated>> woke to: net.schmizz.sshj.userauth.UserAuthException: publickey auth failed
        <<authenticated>> woke to: net.schmizz.sshj.userauth.UserAuthException: publickey auth failed
        <<authenticated>> woke to: net.schmizz.sshj.userauth.UserAuthException: publickey auth failed
        <<authenticated>> woke to: net.schmizz.sshj.userauth.UserAuthException: publickey auth failed
        <<authenticated>> woke to: net.schmizz.sshj.userauth.UserAuthException: publickey auth failed
        <<authenticated>> woke to: net.schmizz.sshj.userauth.UserAuthException: publickey auth failed
        <<authenticated>> woke to: net.schmizz.sshj.userauth.UserAuthException: publickey auth failed
        <<authenticated>> woke to: net.schmizz.sshj.userauth.UserAuthException: publickey auth failed
        <<authenticated>> woke to: net.schmizz.sshj.userauth.UserAuthException: publickey auth failed
        Dying because - java.net.SocketTimeoutException: Read timed out
        Dying because - java.net.SocketTimeoutException: Read timed out
        Dying because - java.net.SocketTimeoutException: Read timed out
        Dying because - java.net.SocketTimeoutException: Read timed out
        Dying because - java.net.SocketTimeoutException: Read timed out
        Dying because - java.net.SocketTimeoutException: Read timed out
        Dying because - java.net.SocketTimeoutException: Read timed out
        Dying because - java.net.SocketTimeoutException: Read timed out
        <<authenticated>> woke to: net.schmizz.sshj.userauth.UserAuthException: publickey auth failed
        Dying because - java.net.SocketTimeoutException: Read timed out
        Dying because - java.net.SocketTimeoutException: Read timed out
        Dying because - java.net.SocketTimeoutException: Read timed out
        Dying because - java.net.SocketTimeoutException: Read timed out
        <<authenticated>> woke to: net.schmizz.sshj.userauth.UserAuthException: publickey auth failed
        Dying because - java.net.SocketTimeoutException: Read timed out
        Dying because - java.net.SocketTimeoutException: Read timed out
        <<authenticated>> woke to: net.schmizz.sshj.userauth.UserAuthException: publickey auth failed
        Dying because - java.net.SocketTimeoutException: Read timed out
        Dying because - java.net.SocketTimeoutException: Read timed out
        <<authenticated>> woke to: net.schmizz.sshj.userauth.UserAuthException: publickey auth failed
        << (ubuntu@50.16.72.110:22) error acquiring SSHClient(ubuntu@50.16.72.110:22): Exhausted available authentication methods
        net.schmizz.sshj.userauth.UserAuthException: Exhausted available authentication methods
        at net.schmizz.sshj.userauth.UserAuthImpl.authenticate(UserAuthImpl.java:114)
        at net.schmizz.sshj.SSHClient.auth(SSHClient.java:204)
        at net.schmizz.sshj.SSHClient.authPublickey(SSHClient.java:304)
        at net.schmizz.sshj.SSHClient.authPublickey(SSHClient.java:323)
        at org.jclouds.sshj.SshjSshClient$1.create(SshjSshClient.java:183)
        at org.jclouds.sshj.SshjSshClient$1.create(SshjSshClient.java:155)
        at org.jclouds.sshj.SshjSshClient.acquire(SshjSshClient.java:204)
        at org.jclouds.sshj.SshjSshClient.connect(SshjSshClient.java:229)
        at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSsh.call(RunScriptOnNodeAsInitScriptUsingSsh.java:107)
        at org.jclouds.compute.strategy.RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:69)

        Show
        Carlos González-Cadenas added a comment - Hello, I'm running your setup and run scripts attached but I cannot successfully launch a cluster on EC2. First I launch in US-EAST-1 a Natty image (ami-fd589594). Then I run the setup script and then the run-test script. After creating the instances, whirr starts giving errors and retrying with no success (see below). Can you help us with this? Thanks Carlos Running configuration script on nodes: [us-east-1/i-c70415a4, us-east-1/i-c50415a6, us-east-1/i-cb0415a8] <<authenticated>> woke to: net.schmizz.sshj.userauth.UserAuthException: publickey auth failed <<authenticated>> woke to: net.schmizz.sshj.userauth.UserAuthException: publickey auth failed <<authenticated>> woke to: net.schmizz.sshj.userauth.UserAuthException: publickey auth failed <<authenticated>> woke to: net.schmizz.sshj.userauth.UserAuthException: publickey auth failed <<authenticated>> woke to: net.schmizz.sshj.userauth.UserAuthException: publickey auth failed <<authenticated>> woke to: net.schmizz.sshj.userauth.UserAuthException: publickey auth failed <<authenticated>> woke to: net.schmizz.sshj.userauth.UserAuthException: publickey auth failed <<authenticated>> woke to: net.schmizz.sshj.userauth.UserAuthException: publickey auth failed <<authenticated>> woke to: net.schmizz.sshj.userauth.UserAuthException: publickey auth failed <<authenticated>> woke to: net.schmizz.sshj.userauth.UserAuthException: publickey auth failed Dying because - java.net.SocketTimeoutException: Read timed out Dying because - java.net.SocketTimeoutException: Read timed out Dying because - java.net.SocketTimeoutException: Read timed out Dying because - java.net.SocketTimeoutException: Read timed out Dying because - java.net.SocketTimeoutException: Read timed out Dying because - java.net.SocketTimeoutException: Read timed out Dying because - java.net.SocketTimeoutException: Read timed out Dying because - java.net.SocketTimeoutException: Read timed out <<authenticated>> woke to: net.schmizz.sshj.userauth.UserAuthException: publickey auth failed Dying because - java.net.SocketTimeoutException: Read timed out Dying because - java.net.SocketTimeoutException: Read timed out Dying because - java.net.SocketTimeoutException: Read timed out Dying because - java.net.SocketTimeoutException: Read timed out <<authenticated>> woke to: net.schmizz.sshj.userauth.UserAuthException: publickey auth failed Dying because - java.net.SocketTimeoutException: Read timed out Dying because - java.net.SocketTimeoutException: Read timed out <<authenticated>> woke to: net.schmizz.sshj.userauth.UserAuthException: publickey auth failed Dying because - java.net.SocketTimeoutException: Read timed out Dying because - java.net.SocketTimeoutException: Read timed out <<authenticated>> woke to: net.schmizz.sshj.userauth.UserAuthException: publickey auth failed << (ubuntu@50.16.72.110:22) error acquiring SSHClient(ubuntu@50.16.72.110:22): Exhausted available authentication methods net.schmizz.sshj.userauth.UserAuthException: Exhausted available authentication methods at net.schmizz.sshj.userauth.UserAuthImpl.authenticate(UserAuthImpl.java:114) at net.schmizz.sshj.SSHClient.auth(SSHClient.java:204) at net.schmizz.sshj.SSHClient.authPublickey(SSHClient.java:304) at net.schmizz.sshj.SSHClient.authPublickey(SSHClient.java:323) at org.jclouds.sshj.SshjSshClient$1.create(SshjSshClient.java:183) at org.jclouds.sshj.SshjSshClient$1.create(SshjSshClient.java:155) at org.jclouds.sshj.SshjSshClient.acquire(SshjSshClient.java:204) at org.jclouds.sshj.SshjSshClient.connect(SshjSshClient.java:229) at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSsh.call(RunScriptOnNodeAsInitScriptUsingSsh.java:107) at org.jclouds.compute.strategy.RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:69)
        Hide
        Christian Mauceri added a comment -

        Hi, because it is quite important to me to launch Whirr from the clouds, I tried on Rackspace and I got a similar error, see below.
        I do not know if it helps but apparently the issue is not restricted to ec2.
        If this is not the place to post such information, please forgive me.
        If I can help, let me know. I'm very frustrated, it is not possible to use Whirr from the clouds.
        Cheers.
        Scripts(ScriptBasedClusterAction.java:215)
           at org.apache.whirr.actions.ScriptBasedClusterAction.doAction(ScriptBasedClusterAction.java:128)
           at org.apache.whirr.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:107)
           at org.apache.whirr.ClusterController.launchCluster(ClusterController.java:109)
           at org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:63)
           at org.apache.whirr.cli.Main.run(Main.java:64)
           at org.apache.whirr.cli.Main.main(Main.java:97)
        Caused by: org.jclouds.rest.AuthorizationException:
        (root:rsa[fingerprint(70:65:c2:67:ba:e4:6d:34:e7:85:36:d3:8f:e0:37:fb),sha1(f6:39:75:1f:08:46:5d:d8:b0:69:a9:0b:ed:a4:0d:2c:32:fd:ad:c8)]@50.56.237.81:22)
        (root:rsa[fingerprint(70:65:c2:67:ba:e4:6d:34:e7:85:36:d3:8f:e0:37:fb),sha1(f6:39:75:1f:08:46:5d:d8:b0:69:a9:0b:ed:a4:0d:2c:32:fd:ad:c8)]@50.56.237.81:22)
        error acquiring SSHClient(timeout=60000): Exhausted available
        authentication methods
           at org.jclouds.sshj.SshjSshClient.propagate(SshjSshClient.java:413)
           at org.jclouds.sshj.SshjSshClient.acquire(SshjSshClient.java:244)
           at org.jclouds.sshj.SshjSshClient.connect(SshjSshClient.java:255)
           at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSsh.call(RunScriptOnNodeAsInitScriptUsingSsh.java:89)
           at org.jclouds.compute.internal.BaseComputeService.runScriptOnNode(BaseComputeService.java:612)
           at org.apache.whirr.actions.ScriptBasedClusterAction$2.call(ScriptBasedClusterAction.java:190)
           at org.apache.whirr.actions.ScriptBasedClusterAction$2.call(ScriptBasedClusterAction.java:178)
           at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
           at java.util.concurrent.FutureTask.run(FutureTask.java:166)
           at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
           at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
           at java.lang.Thread.run(Thread.java:636)
        Caused by: net.schmizz.sshj.userauth.UserAuthException: Exhausted
        available authentication methods
           at net.schmizz.sshj.userauth.UserAuthImpl.authenticate(UserAuthImpl.java:114)
           at net.schmizz.sshj.SSHClient.auth(SSHClient.java:204)
           at net.schmizz.sshj.SSHClient.authPublickey(SSHClient.java:304)
           at net.schmizz.sshj.SSHClient.authPublickey(SSHClient.java:323)
           at org.jclouds.sshj.SshjSshClient$1.create(SshjSshClient.java:199)
           at org.jclouds.sshj.SshjSshClient$1.create(SshjSshClient.java:171)
           at org.jclouds.sshj.SshjSshClient.acquire(SshjSshClient.java:220)
           ... 10 more
        Caused by: net.schmizz.sshj.userauth.UserAuthException: publickey auth failed
           at net.schmizz.sshj.userauth.UserAuthImpl.handle(UserAuthImpl.java:157)
           at net.schmizz.sshj.transport.TransportImpl.handle(TransportImpl.java:474)
           at net.schmizz.sshj.transport.Decoder.decode(Decoder.java:127)
           at net.schmizz.sshj.transport.Decoder.received(Decoder.java:195)
           at net.schmizz.sshj.transport.Reader.run(Reader.java:72)

        Show
        Christian Mauceri added a comment - Hi, because it is quite important to me to launch Whirr from the clouds, I tried on Rackspace and I got a similar error, see below. I do not know if it helps but apparently the issue is not restricted to ec2. If this is not the place to post such information, please forgive me. If I can help, let me know. I'm very frustrated, it is not possible to use Whirr from the clouds. Cheers. Scripts(ScriptBasedClusterAction.java:215)    at org.apache.whirr.actions.ScriptBasedClusterAction.doAction(ScriptBasedClusterAction.java:128)    at org.apache.whirr.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:107)    at org.apache.whirr.ClusterController.launchCluster(ClusterController.java:109)    at org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:63)    at org.apache.whirr.cli.Main.run(Main.java:64)    at org.apache.whirr.cli.Main.main(Main.java:97) Caused by: org.jclouds.rest.AuthorizationException: (root:rsa [fingerprint(70:65:c2:67:ba:e4:6d:34:e7:85:36:d3:8f:e0:37:fb),sha1(f6:39:75:1f:08:46:5d:d8:b0:69:a9:0b:ed:a4:0d:2c:32:fd:ad:c8)] @50.56.237.81:22) (root:rsa [fingerprint(70:65:c2:67:ba:e4:6d:34:e7:85:36:d3:8f:e0:37:fb),sha1(f6:39:75:1f:08:46:5d:d8:b0:69:a9:0b:ed:a4:0d:2c:32:fd:ad:c8)] @50.56.237.81:22) error acquiring SSHClient(timeout=60000): Exhausted available authentication methods    at org.jclouds.sshj.SshjSshClient.propagate(SshjSshClient.java:413)    at org.jclouds.sshj.SshjSshClient.acquire(SshjSshClient.java:244)    at org.jclouds.sshj.SshjSshClient.connect(SshjSshClient.java:255)    at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSsh.call(RunScriptOnNodeAsInitScriptUsingSsh.java:89)    at org.jclouds.compute.internal.BaseComputeService.runScriptOnNode(BaseComputeService.java:612)    at org.apache.whirr.actions.ScriptBasedClusterAction$2.call(ScriptBasedClusterAction.java:190)    at org.apache.whirr.actions.ScriptBasedClusterAction$2.call(ScriptBasedClusterAction.java:178)    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)    at java.util.concurrent.FutureTask.run(FutureTask.java:166)    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)    at java.lang.Thread.run(Thread.java:636) Caused by: net.schmizz.sshj.userauth.UserAuthException: Exhausted available authentication methods    at net.schmizz.sshj.userauth.UserAuthImpl.authenticate(UserAuthImpl.java:114)    at net.schmizz.sshj.SSHClient.auth(SSHClient.java:204)    at net.schmizz.sshj.SSHClient.authPublickey(SSHClient.java:304)    at net.schmizz.sshj.SSHClient.authPublickey(SSHClient.java:323)    at org.jclouds.sshj.SshjSshClient$1.create(SshjSshClient.java:199)    at org.jclouds.sshj.SshjSshClient$1.create(SshjSshClient.java:171)    at org.jclouds.sshj.SshjSshClient.acquire(SshjSshClient.java:220)    ... 10 more Caused by: net.schmizz.sshj.userauth.UserAuthException: publickey auth failed    at net.schmizz.sshj.userauth.UserAuthImpl.handle(UserAuthImpl.java:157)    at net.schmizz.sshj.transport.TransportImpl.handle(TransportImpl.java:474)    at net.schmizz.sshj.transport.Decoder.decode(Decoder.java:127)    at net.schmizz.sshj.transport.Decoder.received(Decoder.java:195)    at net.schmizz.sshj.transport.Reader.run(Reader.java:72)
        Hide
        Andrei Savu added a comment -

        Marking this as a critical issue for 0.8.0.

        Show
        Andrei Savu added a comment - Marking this as a critical issue for 0.8.0.
        Hide
        Andrei Savu added a comment -

        Christian please make sure you are not running Whirr as root.

        Show
        Andrei Savu added a comment - Christian please make sure you are not running Whirr as root.
        Hide
        Christian Mauceri added a comment -

        Thanks Andrei Once again if I can help, let me know.

        Show
        Christian Mauceri added a comment - Thanks Andrei Once again if I can help, let me know.
        Hide
        Christian Mauceri added a comment -

        Andrei I was root I try again.

        Show
        Christian Mauceri added a comment - Andrei I was root I try again.
        Hide
        Christian Mauceri added a comment -

        Ho Andrei, I'm so ashamed and so happy. Thank you so much, it works on Rackspace ! Next step Mahout. Sorry for thr disturbance.

        Show
        Christian Mauceri added a comment - Ho Andrei, I'm so ashamed and so happy. Thank you so much, it works on Rackspace ! Next step Mahout. Sorry for thr disturbance.
        Hide
        Andrei Savu added a comment -

        No worries. Let us know if you find any other issues.

        Show
        Andrei Savu added a comment - No worries. Let us know if you find any other issues.
        Hide
        Michael David added a comment -

        I am seeing the same SSH issue, and I am not running as root. I am running as the default user ec2-user.

        It seems as if whirr is creating a new keypair despite the fact that I already have set one in my properties.

        I've tried a number of AMIs from various tutorials, and can't seem to get it to ssh in.

        whirr version: Whirr 0.5.0-cdh3u2
        hadoop version: Hadoop 0.20.2-cdh3u2

        I've spent hours trying to get to the bottom of this with no luck so far.

        Thanks for any help!

        Properties FILE:
        --------------------
        whirr.cluster-name=myhadoopcluster
        whirr.instance-templates=1 hadoop-jobtracker+hadoop-namenode,1 hadoop-datanode+hadoop-tasktracker
        whirr.provider=aws-ec2
        whirr.identity=<my actual key>
        whirr.credential=<my secret key>
        whirr.login-user=ec2-user
        whirr.hardware-id=m1.small
        whirr.image-id=us-east-1/ami-6936fb00
        whirr.private-key-file=/home/ec2-user/.ssh/id_rsa
        whirr.public-key-file=/home/ec2-user/.ssh/id_rsa.pub
        whirr.hadoop-install-function=install_cdh_hadoop
        whirr.hadoop-configure-function=configure_cdh_hadoop

        LOGS:
        --------------
        [org.apache.whirr.actions.NodeStarter] (pool-5-thread-5) Starting 1 node(s) with roles [hadoop-datanode, hadoop-tasktracker]
        2012-01-03 21:01:08,903 DEBUG [jclouds.compute] (pool-5-thread-5) >> running 1 node group(myhadoopcluster) location(us-east-1) image(us-east-1/ami-6936fb00) hardwareProfile(m1.small) options([groupIds=[], keyPair=null, noKeyPair=false, monitoringEnabled=false, placementGroup=null, noPlacementGroup=false, subnetId=null, userData=null, blockDeviceMappings=[]])
        2012-01-03 21:01:08,904 DEBUG [jclouds.compute] (pool-5-thread-5) >> creating keyPair region(us-east-1) tag(myhadoopcluster)
        2012-01-03 21:01:09,245 DEBUG [jclouds.compute] (pool-5-thread-5) << created keyPair(jclouds#myhadoopcluster#us-east-1#70)
        2012-01-03 21:01:09,246 DEBUG [jclouds.compute] (pool-5-thread-5) >> running 1 instance region(us-east-1) zone(null) ami(ami-6936fb00) params(

        {InstanceType=[m1.small], SecurityGroup.1=[jclouds#myhadoopcluster#us-east-1], KeyName=[jclouds#myhadoopcluster#us-east-1#70]}

        )
        2012-01-03 21:01:09,542 DEBUG [jclouds.compute] (pool-5-thread-5) << started instances(i-xxx)
        2012-01-03 21:01:09,610 DEBUG [jclouds.compute] (pool-5-thread-5) << present instances(i-xxx8)
        2012-01-03 21:01:24,697 DEBUG [jclouds.compute] (user thread 6) >> blocking on socket [address=02.xx.xx.xx, port=22] for 600000 seconds
        2012-01-03 21:01:25,777 DEBUG [jclouds.compute] (user thread 6) << socket [address=01.xx.xx.xx, port=22] opened
        2012-01-03 21:01:48,490 DEBUG [jclouds.compute] (user thread 6) >> blocking on socket [address=02.xx.xx.xx, port=22] for 600000 seconds
        2012-01-03 21:01:48,493 DEBUG [jclouds.compute] (user thread 6) << socket [address=01.xx.xx.xx, port=22] opened
        2012-01-03 21:01:54,292 ERROR [jclouds.compute] (user thread 6) << problem applying options to node(us-east-1/i-901170f2):
        org.jclouds.ssh.SshException: ec2-user@xx.xx.xx.xx: Error connecting to session.
        at org.jclouds.ssh.jsch.JschSshClient.propagate(JschSshClient.java:251)
        at org.jclouds.ssh.jsch.JschSshClient.connect(JschSshClient.java:205)
        at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSsh.call(RunScriptOnNodeAsInitScriptUsingSsh.java:89)
        at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:139)
        at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:55)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:636)
        Caused by: com.jcraft.jsch.JSchException: Auth fail
        at com.jcraft.jsch.Session.connect(Session.java:461)
        at com.jcraft.jsch.Session.connect(Session.java:154)
        at org.jclouds.ssh.jsch.JschSshClient.newSession(JschSshClient.java:246)
        at org.jclouds.ssh.jsch.JschSshClient.connect(JschSshClient.java:185)

        Show
        Michael David added a comment - I am seeing the same SSH issue, and I am not running as root. I am running as the default user ec2-user. It seems as if whirr is creating a new keypair despite the fact that I already have set one in my properties. I've tried a number of AMIs from various tutorials, and can't seem to get it to ssh in. whirr version: Whirr 0.5.0-cdh3u2 hadoop version: Hadoop 0.20.2-cdh3u2 I've spent hours trying to get to the bottom of this with no luck so far. Thanks for any help! Properties FILE: -------------------- whirr.cluster-name=myhadoopcluster whirr.instance-templates=1 hadoop-jobtracker+hadoop-namenode,1 hadoop-datanode+hadoop-tasktracker whirr.provider=aws-ec2 whirr.identity=<my actual key> whirr.credential=<my secret key> whirr.login-user=ec2-user whirr.hardware-id=m1.small whirr.image-id=us-east-1/ami-6936fb00 whirr.private-key-file=/home/ec2-user/.ssh/id_rsa whirr.public-key-file=/home/ec2-user/.ssh/id_rsa.pub whirr.hadoop-install-function=install_cdh_hadoop whirr.hadoop-configure-function=configure_cdh_hadoop LOGS: -------------- [org.apache.whirr.actions.NodeStarter] (pool-5-thread-5) Starting 1 node(s) with roles [hadoop-datanode, hadoop-tasktracker] 2012-01-03 21:01:08,903 DEBUG [jclouds.compute] (pool-5-thread-5) >> running 1 node group(myhadoopcluster) location(us-east-1) image(us-east-1/ami-6936fb00) hardwareProfile(m1.small) options([groupIds=[], keyPair=null, noKeyPair=false, monitoringEnabled=false, placementGroup=null, noPlacementGroup=false, subnetId=null, userData=null, blockDeviceMappings=[]]) 2012-01-03 21:01:08,904 DEBUG [jclouds.compute] (pool-5-thread-5) >> creating keyPair region(us-east-1) tag(myhadoopcluster) 2012-01-03 21:01:09,245 DEBUG [jclouds.compute] (pool-5-thread-5) << created keyPair(jclouds#myhadoopcluster#us-east-1#70) 2012-01-03 21:01:09,246 DEBUG [jclouds.compute] (pool-5-thread-5) >> running 1 instance region(us-east-1) zone(null) ami(ami-6936fb00) params( {InstanceType=[m1.small], SecurityGroup.1=[jclouds#myhadoopcluster#us-east-1], KeyName=[jclouds#myhadoopcluster#us-east-1#70]} ) 2012-01-03 21:01:09,542 DEBUG [jclouds.compute] (pool-5-thread-5) << started instances(i-xxx) 2012-01-03 21:01:09,610 DEBUG [jclouds.compute] (pool-5-thread-5) << present instances(i-xxx8) 2012-01-03 21:01:24,697 DEBUG [jclouds.compute] (user thread 6) >> blocking on socket [address=02.xx.xx.xx, port=22] for 600000 seconds 2012-01-03 21:01:25,777 DEBUG [jclouds.compute] (user thread 6) << socket [address=01.xx.xx.xx, port=22] opened 2012-01-03 21:01:48,490 DEBUG [jclouds.compute] (user thread 6) >> blocking on socket [address=02.xx.xx.xx, port=22] for 600000 seconds 2012-01-03 21:01:48,493 DEBUG [jclouds.compute] (user thread 6) << socket [address=01.xx.xx.xx, port=22] opened 2012-01-03 21:01:54,292 ERROR [jclouds.compute] (user thread 6) << problem applying options to node(us-east-1/i-901170f2): org.jclouds.ssh.SshException: ec2-user@xx.xx.xx.xx: Error connecting to session. at org.jclouds.ssh.jsch.JschSshClient.propagate(JschSshClient.java:251) at org.jclouds.ssh.jsch.JschSshClient.connect(JschSshClient.java:205) at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSsh.call(RunScriptOnNodeAsInitScriptUsingSsh.java:89) at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:139) at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:55) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) Caused by: com.jcraft.jsch.JSchException: Auth fail at com.jcraft.jsch.Session.connect(Session.java:461) at com.jcraft.jsch.Session.connect(Session.java:154) at org.jclouds.ssh.jsch.JschSshClient.newSession(JschSshClient.java:246) at org.jclouds.ssh.jsch.JschSshClient.connect(JschSshClient.java:185)
        Hide
        Andrei Savu added a comment -

        Michael try to add:

        whirr.cluster-user=something-different-than-ec2-user
        

        You should be able to login using:

        ssh -i /home/ec2-user/.ssh/id_rsa whirr.cluster-user@host
        

        I am trying this workaround now.

        I will create soon a patch to address the following issues:

        • fail fast when running Whirr as root
        • do not fail if a remote user with the same name as whirr.cluster-user already exists
        Show
        Andrei Savu added a comment - Michael try to add: whirr.cluster-user=something-different-than-ec2-user You should be able to login using: ssh -i /home/ec2-user/.ssh/id_rsa whirr.cluster-user@host I am trying this workaround now. I will create soon a patch to address the following issues: fail fast when running Whirr as root do not fail if a remote user with the same name as whirr.cluster-user already exists
        Hide
        Michael David added a comment -

        Thanks for this. I will give it a go. I thought that Whirr does NOT create a new user on the remote server. Does it?

        Show
        Michael David added a comment - Thanks for this. I will give it a go. I thought that Whirr does NOT create a new user on the remote server. Does it?
        Hide
        Andrei Savu added a comment -

        I'm not 100% sure for 0.5.0 but this is true for 0.6+. BTW you should upgrade to our latest stable release (0.7.0). See: http://www.apache.org/dyn/closer.cgi/whirr/

        Show
        Andrei Savu added a comment - I'm not 100% sure for 0.5.0 but this is true for 0.6+. BTW you should upgrade to our latest stable release (0.7.0). See: http://www.apache.org/dyn/closer.cgi/whirr/
        Hide
        Michael David added a comment - - edited

        Hi Andrei - Just tried with 0.7.0 and am getting new errors:

        Instead of using the properties file, I tried to invoke keys inline instead like the tuorial here: http://whirr.apache.org/docs/0.7.0/whirr-in-5-minutes.html

        ./whirr launch-cluster --config hadoop.properties --private-key-file ~/.ssh/id_rsa_whirr --public-key-file ~/.ssh/id_rsa_whirr.pub

        It seems that I'm getting a different set of auth errors:

        Dying because - net.schmizz.sshj.transport.TransportException: Broken transport; encountered EOF
        Dying because - net.schmizz.sshj.transport.TransportException: Broken transport; encountered EOF
        <<kex done>> woke to: net.schmizz.sshj.transport.TransportException: Broken transport; encountered EOF
        << (USER:rsa[fingerprint(FINGERPRINT),sha1(SHA1)]@xx.xx.xx.xx:22) error acquiring SSHClient(timeout=60000): Broken transport; encountered EOF
        net.schmizz.sshj.transport.TransportException: Broken transport; encountered EOF
        at net.schmizz.sshj.transport.Reader.run(Reader.java:70)
        <<authenticated>> woke to: net.schmizz.sshj.userauth.UserAuthException: publickey auth failed
        << (USER:rsa[fingerprint(FINGERPRINT),sha1(SHA1)]@xx.xx.xx:22) error acquiring SSHClient(timeout=60000): exception not retryable
        net.schmizz.sshj.userauth.UserAuthException: Exhausted available authentication methods
        at net.schmizz.sshj.userauth.UserAuthImpl.authenticate(UserAuthImpl.java:114)
        at net.schmizz.sshj.SSHClient.auth(SSHClient.java:204)
        at net.schmizz.sshj.SSHClient.authPublickey(SSHClient.java:304)
        at net.schmizz.sshj.SSHClient.authPublickey(SSHClient.java:323)
        at org.jclouds.sshj.SshjSshClient$1.create(SshjSshClient.java:199)
        at org.jclouds.sshj.SshjSshClient$1.create(SshjSshClient.java:171)
        at org.jclouds.sshj.SshjSshClient.acquire(SshjSshClient.java:220)
        at org.jclouds.sshj.SshjSshClient.connect(SshjSshClient.java:255)
        at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSsh.call(RunScriptOnNodeAsInitScriptUsingSsh.java:89)
        at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:150)
        at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:57)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:636)
        Caused by: net.schmizz.sshj.userauth.UserAuthException: publickey auth failed
        at net.schmizz.sshj.userauth.UserAuthImpl.handle(UserAuthImpl.java:157)
        at net.schmizz.sshj.transport.TransportImpl.handle(TransportImpl.java:474)
        at net.schmizz.sshj.transport.Decoder.decode(Decoder.java:127)
        at net.schmizz.sshj.transport.Decoder.received(Decoder.java:195)
        at net.schmizz.sshj.transport.Reader.run(Reader.java:72)

        << problem applying options to node(us-east-1/i-xxxxxe):
        org.jclouds.rest.AuthorizationException: (USER:rsa[fingerprint(RSA),sha1(SHA)]@xx.xx.xx:22) (USER:rsa[fingerprint(RSA),sha1(SHA1)]@xx.xx.xx:22) error acquiring SSHClient(timeout=60000): Exhausted available authentication methods
        at org.jclouds.sshj.SshjSshClient.propagate(SshjSshClient.java:413)
        at org.jclouds.sshj.SshjSshClient.acquire(SshjSshClient.java:244)
        at org.jclouds.sshj.SshjSshClient.connect(SshjSshClient.java:255)
        at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSsh.call(RunScriptOnNodeAsInitScriptUsingSsh.java:89)
        at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:150)
        at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:57)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:636)
        Caused by: net.schmizz.sshj.userauth.UserAuthException: Exhausted available authentication methods
        at net.schmizz.sshj.userauth.UserAuthImpl.authenticate(UserAuthImpl.java:114)
        at net.schmizz.sshj.SSHClient.auth(SSHClient.java:204)
        at net.schmizz.sshj.SSHClient.authPublickey(SSHClient.java:304)
        at net.schmizz.sshj.SSHClient.authPublickey(SSHClient.java:323)
        at org.jclouds.sshj.SshjSshClient$1.create(SshjSshClient.java:199)
        at org.jclouds.sshj.SshjSshClient$1.create(SshjSshClient.java:171)
        at org.jclouds.sshj.SshjSshClient.acquire(SshjSshClient.java:220)
        ... 9 more
        Caused by: net.schmizz.sshj.userauth.UserAuthException: publickey auth failed
        at net.schmizz.sshj.userauth.UserAuthImpl.handle(UserAuthImpl.java:157)
        at net.schmizz.sshj.transport.TransportImpl.handle(TransportImpl.java:474)
        at net.schmizz.sshj.transport.Decoder.decode(Decoder.java:127)
        at net.schmizz.sshj.transport.Decoder.received(Decoder.java:195)
        at net.schmizz.sshj.transport.Reader.run(Reader.java:72)
        <<authenticated>> woke to: net.schmizz.sshj.userauth.UserAuthException: publickey auth failed
        << (USER:rsa[fingerprint(RSA),sha1(SHA1)]@xx.xxx.xx:22) error acquiring SSHClient(timeout=60000): exception not retryable
        net.schmizz.sshj.userauth.UserAuthException: Exhausted available authentication methods
        at net.schmizz.sshj.userauth.UserAuthImpl.authenticate(UserAuthImpl.java:114)
        at net.schmizz.sshj.SSHClient.auth(SSHClient.java:204)
        at net.schmizz.sshj.SSHClient.authPublickey(SSHClient.java:304)
        at net.schmizz.sshj.SSHClient.authPublickey(SSHClient.java:323)
        at org.jclouds.sshj.SshjSshClient$1.create(SshjSshClient.java:199)
        at org.jclouds.sshj.SshjSshClient$1.create(SshjSshClient.java:171)
        at org.jclouds.sshj.SshjSshClient.acquire(SshjSshClient.java:220)
        at org.jclouds.sshj.SshjSshClient.connect(SshjSshClient.java:255)
        at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSsh.call(RunScriptOnNodeAsInitScriptUsingSsh.java:89)
        at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:150)
        at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:57)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:636)
        Caused by: net.schmizz.sshj.userauth.UserAuthException: publickey auth failed
        at net.schmizz.sshj.userauth.UserAuthImpl.handle(UserAuthImpl.java:157)
        at net.schmizz.sshj.transport.TransportImpl.handle(TransportImpl.java:474)
        at net.schmizz.sshj.transport.Decoder.decode(Decoder.java:127)
        at net.schmizz.sshj.transport.Decoder.received(Decoder.java:195)
        at net.schmizz.sshj.transport.Reader.run(Reader.java:72)
        << (USER:rsa[fingerprint(RSA),sha1(SHA1)]@xx.xx.xx:22) error acquiring SSHClient(timeout=60000): Exhausted available authentication methods
        net.schmizz.sshj.userauth.UserAuthException: Exhausted available authentication methods
        at net.schmizz.sshj.userauth.UserAuthImpl.authenticate(UserAuthImpl.java:114)
        at net.schmizz.sshj.SSHClient.auth(SSHClient.java:204)
        at net.schmizz.sshj.SSHClient.authPublickey(SSHClient.java:304)
        at net.schmizz.sshj.SSHClient.authPublickey(SSHClient.java:323)
        at org.jclouds.sshj.SshjSshClient$1.create(SshjSshClient.java:199)
        at org.jclouds.sshj.SshjSshClient$1.create(SshjSshClient.java:171)
        at org.jclouds.sshj.SshjSshClient.acquire(SshjSshClient.java:220)
        at org.jclouds.sshj.SshjSshClient.connect(SshjSshClient.java:255)
        at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSsh.call(RunScriptOnNodeAsInitScriptUsingSsh.java:89)
        at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:150)
        at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:57)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:636)
        Caused by: net.schmizz.sshj.userauth.UserAuthException: publickey auth failed
        at net.schmizz.sshj.userauth.UserAuthImpl.handle(UserAuthImpl.java:157)
        at net.schmizz.sshj.transport.TransportImpl.handle(TransportImpl.java:474)
        at net.schmizz.sshj.transport.Decoder.decode(Decoder.java:127)
        at net.schmizz.sshj.transport.Decoder.received(Decoder.java:195)
        at net.schmizz.sshj.transport.Reader.run(Reader.java:72)

        Thanks - really appreciate your speedy answers here!

        Show
        Michael David added a comment - - edited Hi Andrei - Just tried with 0.7.0 and am getting new errors: Instead of using the properties file, I tried to invoke keys inline instead like the tuorial here: http://whirr.apache.org/docs/0.7.0/whirr-in-5-minutes.html ./whirr launch-cluster --config hadoop.properties --private-key-file ~/.ssh/id_rsa_whirr --public-key-file ~/.ssh/id_rsa_whirr.pub It seems that I'm getting a different set of auth errors: Dying because - net.schmizz.sshj.transport.TransportException: Broken transport; encountered EOF Dying because - net.schmizz.sshj.transport.TransportException: Broken transport; encountered EOF <<kex done>> woke to: net.schmizz.sshj.transport.TransportException: Broken transport; encountered EOF << (USER:rsa [fingerprint(FINGERPRINT),sha1(SHA1)] @xx.xx.xx.xx:22) error acquiring SSHClient(timeout=60000): Broken transport; encountered EOF net.schmizz.sshj.transport.TransportException: Broken transport; encountered EOF at net.schmizz.sshj.transport.Reader.run(Reader.java:70) <<authenticated>> woke to: net.schmizz.sshj.userauth.UserAuthException: publickey auth failed << (USER:rsa [fingerprint(FINGERPRINT),sha1(SHA1)] @xx.xx.xx:22) error acquiring SSHClient(timeout=60000): exception not retryable net.schmizz.sshj.userauth.UserAuthException: Exhausted available authentication methods at net.schmizz.sshj.userauth.UserAuthImpl.authenticate(UserAuthImpl.java:114) at net.schmizz.sshj.SSHClient.auth(SSHClient.java:204) at net.schmizz.sshj.SSHClient.authPublickey(SSHClient.java:304) at net.schmizz.sshj.SSHClient.authPublickey(SSHClient.java:323) at org.jclouds.sshj.SshjSshClient$1.create(SshjSshClient.java:199) at org.jclouds.sshj.SshjSshClient$1.create(SshjSshClient.java:171) at org.jclouds.sshj.SshjSshClient.acquire(SshjSshClient.java:220) at org.jclouds.sshj.SshjSshClient.connect(SshjSshClient.java:255) at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSsh.call(RunScriptOnNodeAsInitScriptUsingSsh.java:89) at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:150) at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:57) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) Caused by: net.schmizz.sshj.userauth.UserAuthException: publickey auth failed at net.schmizz.sshj.userauth.UserAuthImpl.handle(UserAuthImpl.java:157) at net.schmizz.sshj.transport.TransportImpl.handle(TransportImpl.java:474) at net.schmizz.sshj.transport.Decoder.decode(Decoder.java:127) at net.schmizz.sshj.transport.Decoder.received(Decoder.java:195) at net.schmizz.sshj.transport.Reader.run(Reader.java:72) << problem applying options to node(us-east-1/i-xxxxxe): org.jclouds.rest.AuthorizationException: (USER:rsa [fingerprint(RSA),sha1(SHA)] @xx.xx.xx:22) (USER:rsa [fingerprint(RSA),sha1(SHA1)] @xx.xx.xx:22) error acquiring SSHClient(timeout=60000): Exhausted available authentication methods at org.jclouds.sshj.SshjSshClient.propagate(SshjSshClient.java:413) at org.jclouds.sshj.SshjSshClient.acquire(SshjSshClient.java:244) at org.jclouds.sshj.SshjSshClient.connect(SshjSshClient.java:255) at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSsh.call(RunScriptOnNodeAsInitScriptUsingSsh.java:89) at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:150) at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:57) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) Caused by: net.schmizz.sshj.userauth.UserAuthException: Exhausted available authentication methods at net.schmizz.sshj.userauth.UserAuthImpl.authenticate(UserAuthImpl.java:114) at net.schmizz.sshj.SSHClient.auth(SSHClient.java:204) at net.schmizz.sshj.SSHClient.authPublickey(SSHClient.java:304) at net.schmizz.sshj.SSHClient.authPublickey(SSHClient.java:323) at org.jclouds.sshj.SshjSshClient$1.create(SshjSshClient.java:199) at org.jclouds.sshj.SshjSshClient$1.create(SshjSshClient.java:171) at org.jclouds.sshj.SshjSshClient.acquire(SshjSshClient.java:220) ... 9 more Caused by: net.schmizz.sshj.userauth.UserAuthException: publickey auth failed at net.schmizz.sshj.userauth.UserAuthImpl.handle(UserAuthImpl.java:157) at net.schmizz.sshj.transport.TransportImpl.handle(TransportImpl.java:474) at net.schmizz.sshj.transport.Decoder.decode(Decoder.java:127) at net.schmizz.sshj.transport.Decoder.received(Decoder.java:195) at net.schmizz.sshj.transport.Reader.run(Reader.java:72) <<authenticated>> woke to: net.schmizz.sshj.userauth.UserAuthException: publickey auth failed << (USER:rsa [fingerprint(RSA),sha1(SHA1)] @xx.xxx.xx:22) error acquiring SSHClient(timeout=60000): exception not retryable net.schmizz.sshj.userauth.UserAuthException: Exhausted available authentication methods at net.schmizz.sshj.userauth.UserAuthImpl.authenticate(UserAuthImpl.java:114) at net.schmizz.sshj.SSHClient.auth(SSHClient.java:204) at net.schmizz.sshj.SSHClient.authPublickey(SSHClient.java:304) at net.schmizz.sshj.SSHClient.authPublickey(SSHClient.java:323) at org.jclouds.sshj.SshjSshClient$1.create(SshjSshClient.java:199) at org.jclouds.sshj.SshjSshClient$1.create(SshjSshClient.java:171) at org.jclouds.sshj.SshjSshClient.acquire(SshjSshClient.java:220) at org.jclouds.sshj.SshjSshClient.connect(SshjSshClient.java:255) at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSsh.call(RunScriptOnNodeAsInitScriptUsingSsh.java:89) at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:150) at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:57) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) Caused by: net.schmizz.sshj.userauth.UserAuthException: publickey auth failed at net.schmizz.sshj.userauth.UserAuthImpl.handle(UserAuthImpl.java:157) at net.schmizz.sshj.transport.TransportImpl.handle(TransportImpl.java:474) at net.schmizz.sshj.transport.Decoder.decode(Decoder.java:127) at net.schmizz.sshj.transport.Decoder.received(Decoder.java:195) at net.schmizz.sshj.transport.Reader.run(Reader.java:72) << (USER:rsa [fingerprint(RSA),sha1(SHA1)] @xx.xx.xx:22) error acquiring SSHClient(timeout=60000): Exhausted available authentication methods net.schmizz.sshj.userauth.UserAuthException: Exhausted available authentication methods at net.schmizz.sshj.userauth.UserAuthImpl.authenticate(UserAuthImpl.java:114) at net.schmizz.sshj.SSHClient.auth(SSHClient.java:204) at net.schmizz.sshj.SSHClient.authPublickey(SSHClient.java:304) at net.schmizz.sshj.SSHClient.authPublickey(SSHClient.java:323) at org.jclouds.sshj.SshjSshClient$1.create(SshjSshClient.java:199) at org.jclouds.sshj.SshjSshClient$1.create(SshjSshClient.java:171) at org.jclouds.sshj.SshjSshClient.acquire(SshjSshClient.java:220) at org.jclouds.sshj.SshjSshClient.connect(SshjSshClient.java:255) at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSsh.call(RunScriptOnNodeAsInitScriptUsingSsh.java:89) at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:150) at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:57) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) Caused by: net.schmizz.sshj.userauth.UserAuthException: publickey auth failed at net.schmizz.sshj.userauth.UserAuthImpl.handle(UserAuthImpl.java:157) at net.schmizz.sshj.transport.TransportImpl.handle(TransportImpl.java:474) at net.schmizz.sshj.transport.Decoder.decode(Decoder.java:127) at net.schmizz.sshj.transport.Decoder.received(Decoder.java:195) at net.schmizz.sshj.transport.Reader.run(Reader.java:72) Thanks - really appreciate your speedy answers here!
        Hide
        Joris Poort added a comment -

        Still having issues with this too here...

        I was able to fix it in os/x through recreating keys but now having issues again on ubuntu vm on amazon. Regenerated keys etc. but still getting this error.

        Right now using:

        whirr.bootstrap-user=ubuntu
        whirr.cluster-user=ubuntu

        Andrei - anything else I can try?

        Show
        Joris Poort added a comment - Still having issues with this too here... I was able to fix it in os/x through recreating keys but now having issues again on ubuntu vm on amazon. Regenerated keys etc. but still getting this error. Right now using: whirr.bootstrap-user=ubuntu whirr.cluster-user=ubuntu Andrei - anything else I can try?
        Hide
        Andrei Savu added a comment -

        whirr.bootstrap-user should be != than whirr.cluster-user

        Unless you are using a custom AMI most of the time you don't need to specify a value for whirr.bootstrap-user.

        Important whirr.cluster-user should not already exist on the remote machine. Try something like whirr.cluster-user=whirr or something else unique.

        Show
        Andrei Savu added a comment - whirr.bootstrap-user should be != than whirr.cluster-user Unless you are using a custom AMI most of the time you don't need to specify a value for whirr.bootstrap-user. Important whirr.cluster-user should not already exist on the remote machine. Try something like whirr.cluster-user=whirr or something else unique.
        Hide
        Joris Poort added a comment -

        Thanks Andrei - I was using a custom AMI but your explanation/tips helped fix it, really appreciate it .

        Joris

        Show
        Joris Poort added a comment - Thanks Andrei - I was using a custom AMI but your explanation/tips helped fix it, really appreciate it . Joris
        Hide
        Andrei Savu added a comment -

        Joris I'm happy it works for you. We are planning to address this unexpected behaviour in 0.7.1.

        Show
        Andrei Savu added a comment - Joris I'm happy it works for you. We are planning to address this unexpected behaviour in 0.7.1.
        Hide
        Andrei Savu added a comment -

        I will close this as invalid because most of the time this false problem is generated by invalid configuration.

        Show
        Andrei Savu added a comment - I will close this as invalid because most of the time this false problem is generated by invalid configuration.
        Hide
        Joris Poort added a comment -

        Having some issues with this again - not sure if its from 0.7.1 but we just transitioned to this latest version.

        2012-03-01 22:54:24,843 ERROR [net.schmizz.sshj.transport.TransportImpl] (reader) Dying because - java.net.SocketTimeoutException: Read timed out
        2012-03-01 22:54:30,986 ERROR [net.schmizz.sshj.transport.TransportImpl] (reader) Dying because - java.net.SocketTimeoutException: Read timed out

        I started with a working configuration (standard Ubuntu 10.04) then created a custom AMI (without changing anything) and tried to boot the cluster.

        I'm using:
        whirr.bootstrap-user=ubuntu
        whirr.cluster-user=whirr
        jclouds.ec2.ami-query=owner-id=[ID]

        Any ideas on what I can try out?

        Thanks in advance!

        Show
        Joris Poort added a comment - Having some issues with this again - not sure if its from 0.7.1 but we just transitioned to this latest version. 2012-03-01 22:54:24,843 ERROR [net.schmizz.sshj.transport.TransportImpl] (reader) Dying because - java.net.SocketTimeoutException: Read timed out 2012-03-01 22:54:30,986 ERROR [net.schmizz.sshj.transport.TransportImpl] (reader) Dying because - java.net.SocketTimeoutException: Read timed out I started with a working configuration (standard Ubuntu 10.04) then created a custom AMI (without changing anything) and tried to boot the cluster. I'm using: whirr.bootstrap-user=ubuntu whirr.cluster-user=whirr jclouds.ec2.ami-query=owner-id= [ID] Any ideas on what I can try out? Thanks in advance!
        Hide
        Andrei Savu added a comment - - edited

        whirr.bootstrap-user is going to be available in 0.8.0. For 0.7.1 you have to use whirr.login-user. Is the cluster deployment failing? (sometimes errors like this can be misleading) Also make sure you are not using t1.micro.

        Show
        Andrei Savu added a comment - - edited whirr.bootstrap-user is going to be available in 0.8.0. For 0.7.1 you have to use whirr.login-user. Is the cluster deployment failing? (sometimes errors like this can be misleading) Also make sure you are not using t1.micro.
        Hide
        Joris Poort added a comment -

        I've tried several sizes but all fail to deploy successfully. Its starting the nodes in EC2, but never getting past the setup (just hangs after the following):

        Bootstrapping cluster
        Configuring template
        Configuring template
        Starting 1 node(s) with roles [hadoop-datanode, hadoop-tasktracker]
        Starting 1 node(s) with roles [hadoop-namenode, hadoop-jobtracker]

        This is the first time (in my experience) that it just hangs w/o giving more stack trace errors.

        Show
        Joris Poort added a comment - I've tried several sizes but all fail to deploy successfully. Its starting the nodes in EC2, but never getting past the setup (just hangs after the following): Bootstrapping cluster Configuring template Configuring template Starting 1 node(s) with roles [hadoop-datanode, hadoop-tasktracker] Starting 1 node(s) with roles [hadoop-namenode, hadoop-jobtracker] This is the first time (in my experience) that it just hangs w/o giving more stack trace errors.
        Hide
        Joris Poort added a comment - - edited

        Seems like it might have been defining "whirr.bootstrap-user" instead of "whirr.login-user" caused the issue. Seems odd, but right now I've tracked it down to the following: custom AMI (standard OS, private or public) in combination with definition of either ("bootstrap-user" or no "login-user" - not sure which one) specified causes the issue.

        Ref: https://issues.apache.org/jira/browse/WHIRR-475

        Show
        Joris Poort added a comment - - edited Seems like it might have been defining "whirr.bootstrap-user" instead of "whirr.login-user" caused the issue. Seems odd, but right now I've tracked it down to the following: custom AMI (standard OS, private or public) in combination with definition of either ("bootstrap-user" or no "login-user" - not sure which one) specified causes the issue. Ref: https://issues.apache.org/jira/browse/WHIRR-475
        Hide
        Andrew Ash added a comment -

        I'm still seeing this error with these configuration settings:

        # hadoop.properties
        whirr.cluster-name=myhadoopcluster
        whirr.instance-templates=1 hadoop-jobtracker+hadoop-namenode,1 hadoop-datanode+hadoop-tasktracker
        whirr.provider=aws-ec2
        whirr.identity=<my identity>
        whirr.credential=<my credentials>
        whirr.cluster-user=whirr
        whirr.private-key-file=/home/<my user>/.ssh/id_rsa
        whirr.public-key-file=/home/<my user>/.ssh/id_rsa.pub
        whirr.env.repo=cdh4
        whirr.hadoop.install-function=install_cdh_hadoop
        whirr.hadoop.configure-function=configure_cdh_hadoop
        whirr.hardware-id=m1.large
        whirr.image-id=eu-west-1/ami-5fd2d12b
        whirr.location-id=eu-west-1
        

        And this version of whirr:

        $ whirr version
        Apache Whirr 0.8.0-cdh4.1.3
        jclouds 1.5.0-beta.11
        $
        

        It appears in the logs that whirr is continuing to create a new keypair and initializing that into the new VM, but then attempting to use the keypair I defined. Here are those entries in the log:

        2013-02-15 22:26:29,326 DEBUG [jclouds.compute] (main) >> searching params({imageId=eu-west-1/ami-5fd2d12b, locationId=eu-west-1, hardwareId=m1.large})
        2013-02-15 22:26:29,327 DEBUG [jclouds.compute] (main) <<   matched image(eu-west-1/ami-5fd2d12b) hardware(m1.large) location(eu-west-1)
        2013-02-15 22:26:29,333 INFO  [org.apache.whirr.compute.NodeStarter] (pool-3-thread-4) Starting 1 node(s) with roles [hadoop-jobtracker, hadoop-namenode]
        2013-02-15 22:26:29,335 DEBUG [jclouds.compute] (pool-3-thread-4) >> running 1 node group(myhadoopcluster) location(eu-west-1) image(eu-west-1/ami-5fd2d12b) hardwareProfile(m1.large) options({scriptPresent=true, userData=[35, 99, 108, ... <more user data> ...], blockDeviceMappings=[[deviceName=/dev/sdc, virtualName=ephemeral1, snapshotId=null, sizeInGib=null, noDevice=null, deleteOnTermination=null]]})
        2013-02-15 22:26:30,469 DEBUG [jclouds.compute] (pool-3-thread-2) << created keyPair([region=eu-west-1, keyName=jclouds#myhadoopcluster#68e, fingerprint=49:57:db:e8:30:57:94:4a:3b:7c:b6:db:5f:d4:8d:9a,sha1OfPrivateKey=d9:6b:2d:f3:97:e8:97:ed:d8:de:f3:8a:c1:9b:53:0a:7f:18:cc:77, keyMaterial?=true])
        2013-02-15 22:26:30,470 DEBUG [jclouds.compute] (pool-3-thread-2) >> creating securityGroup region(eu-west-1) name(jclouds#myhadoopcluster)
        2013-02-15 22:26:30,646 DEBUG [jclouds.compute] (pool-3-thread-2) << created securityGroup(jclouds#myhadoopcluster)
        2013-02-15 22:26:30,646 DEBUG [jclouds.compute] (pool-3-thread-2) >> authorizing securityGroup region(eu-west-1) name(jclouds#myhadoopcluster) port(22)
        2013-02-15 22:26:30,728 DEBUG [jclouds.compute] (pool-3-thread-2) << authorized securityGroup(jclouds#myhadoopcluster)
        2013-02-15 22:26:30,728 DEBUG [jclouds.compute] (pool-3-thread-2) >> authorizing securityGroup region(eu-west-1) name(jclouds#myhadoopcluster) permission to itself
        2013-02-15 22:26:30,927 DEBUG [jclouds.compute] (pool-3-thread-2) << authorized securityGroup(jclouds#myhadoopcluster)
        2013-02-15 22:26:30,928 DEBUG [jclouds.compute] (pool-3-thread-2) >> running 1 instance region(eu-west-1) zone(null) ami(ami-5fd2d12b) params({InstanceType=[m1.large], SecurityGroup.1=[jclouds#myhadoopcluster], KeyName=[jclouds#myhadoopcluster#68e], UserData=[<some user data>], BlockDeviceMapping.1.DeviceName=[/dev/sdc], BlockDeviceMapping.1.VirtualName=[ephemeral1]})
        2013-02-15 22:26:30,930 DEBUG [jclouds.compute] (pool-3-thread-4) >> running 1 instance region(eu-west-1) zone(null) ami(ami-5fd2d12b) params({InstanceType=[m1.large], SecurityGroup.1=[jclouds#myhadoopcluster], KeyName=[jclouds#myhadoopcluster#68e], UserData=[<some user data>], BlockDeviceMapping.1.DeviceName=[/dev/sdc], BlockDeviceMapping.1.VirtualName=[ephemeral1]})
        

        Note the "created keyPair" line and the running instance with that parameter (68e is the id of the generated key).

        Is there a configuration setting I can use to force whirr to use my given ssh keypair?

        Show
        Andrew Ash added a comment - I'm still seeing this error with these configuration settings: # hadoop.properties whirr.cluster-name=myhadoopcluster whirr.instance-templates=1 hadoop-jobtracker+hadoop-namenode,1 hadoop-datanode+hadoop-tasktracker whirr.provider=aws-ec2 whirr.identity=<my identity> whirr.credential=<my credentials> whirr.cluster-user=whirr whirr.private-key-file=/home/<my user>/.ssh/id_rsa whirr.public-key-file=/home/<my user>/.ssh/id_rsa.pub whirr.env.repo=cdh4 whirr.hadoop.install-function=install_cdh_hadoop whirr.hadoop.configure-function=configure_cdh_hadoop whirr.hardware-id=m1.large whirr.image-id=eu-west-1/ami-5fd2d12b whirr.location-id=eu-west-1 And this version of whirr: $ whirr version Apache Whirr 0.8.0-cdh4.1.3 jclouds 1.5.0-beta.11 $ It appears in the logs that whirr is continuing to create a new keypair and initializing that into the new VM, but then attempting to use the keypair I defined. Here are those entries in the log: 2013-02-15 22:26:29,326 DEBUG [jclouds.compute] (main) >> searching params({imageId=eu-west-1/ami-5fd2d12b, locationId=eu-west-1, hardwareId=m1.large}) 2013-02-15 22:26:29,327 DEBUG [jclouds.compute] (main) << matched image(eu-west-1/ami-5fd2d12b) hardware(m1.large) location(eu-west-1) 2013-02-15 22:26:29,333 INFO [org.apache.whirr.compute.NodeStarter] (pool-3-thread-4) Starting 1 node(s) with roles [hadoop-jobtracker, hadoop-namenode] 2013-02-15 22:26:29,335 DEBUG [jclouds.compute] (pool-3-thread-4) >> running 1 node group(myhadoopcluster) location(eu-west-1) image(eu-west-1/ami-5fd2d12b) hardwareProfile(m1.large) options({scriptPresent=true, userData=[35, 99, 108, ... <more user data> ...], blockDeviceMappings=[[deviceName=/dev/sdc, virtualName=ephemeral1, snapshotId=null, sizeInGib=null, noDevice=null, deleteOnTermination=null]]}) 2013-02-15 22:26:30,469 DEBUG [jclouds.compute] (pool-3-thread-2) << created keyPair([region=eu-west-1, keyName=jclouds#myhadoopcluster#68e, fingerprint=49:57:db:e8:30:57:94:4a:3b:7c:b6:db:5f:d4:8d:9a,sha1OfPrivateKey=d9:6b:2d:f3:97:e8:97:ed:d8:de:f3:8a:c1:9b:53:0a:7f:18:cc:77, keyMaterial?=true]) 2013-02-15 22:26:30,470 DEBUG [jclouds.compute] (pool-3-thread-2) >> creating securityGroup region(eu-west-1) name(jclouds#myhadoopcluster) 2013-02-15 22:26:30,646 DEBUG [jclouds.compute] (pool-3-thread-2) << created securityGroup(jclouds#myhadoopcluster) 2013-02-15 22:26:30,646 DEBUG [jclouds.compute] (pool-3-thread-2) >> authorizing securityGroup region(eu-west-1) name(jclouds#myhadoopcluster) port(22) 2013-02-15 22:26:30,728 DEBUG [jclouds.compute] (pool-3-thread-2) << authorized securityGroup(jclouds#myhadoopcluster) 2013-02-15 22:26:30,728 DEBUG [jclouds.compute] (pool-3-thread-2) >> authorizing securityGroup region(eu-west-1) name(jclouds#myhadoopcluster) permission to itself 2013-02-15 22:26:30,927 DEBUG [jclouds.compute] (pool-3-thread-2) << authorized securityGroup(jclouds#myhadoopcluster) 2013-02-15 22:26:30,928 DEBUG [jclouds.compute] (pool-3-thread-2) >> running 1 instance region(eu-west-1) zone(null) ami(ami-5fd2d12b) params({InstanceType=[m1.large], SecurityGroup.1=[jclouds#myhadoopcluster], KeyName=[jclouds#myhadoopcluster#68e], UserData=[<some user data>], BlockDeviceMapping.1.DeviceName=[/dev/sdc], BlockDeviceMapping.1.VirtualName=[ephemeral1]}) 2013-02-15 22:26:30,930 DEBUG [jclouds.compute] (pool-3-thread-4) >> running 1 instance region(eu-west-1) zone(null) ami(ami-5fd2d12b) params({InstanceType=[m1.large], SecurityGroup.1=[jclouds#myhadoopcluster], KeyName=[jclouds#myhadoopcluster#68e], UserData=[<some user data>], BlockDeviceMapping.1.DeviceName=[/dev/sdc], BlockDeviceMapping.1.VirtualName=[ephemeral1]}) Note the "created keyPair" line and the running instance with that parameter (68e is the id of the generated key). Is there a configuration setting I can use to force whirr to use my given ssh keypair?
        Hide
        nareshparimi added a comment - - edited

        I have solved this issue:---

        I have stuck with this issue and googled a lot for hints ,But finally i have solved this issue myself . Hope this post is useful.

        Root cause for this problem : The non existence of the user, we are trying to runscripts on VM side.

        Solution: Make sure the user we are using is existing and check whether we are able to do SSH with that user onto VM machine.

        ERROR LOG:

        initializing {id=openstack-nova, name=OpenStack Nova Diablo+ API, views=[org.jclouds.compute.ComputeServiceContext], endpointName=KeyStone base url ending in /v2.0/, identityName=$

        {tenantName}

        :$

        {userName} or ${userName}

        , if your keystone supports a default tenant, credentialName=Optional.of($

        {password}

        ), documentation=http://api.openstack.org/, api=interface org.jclouds.openstack.nova.v2_0.NovaApi, asyncApi=interface org.jclouds.openstack.nova.v2_0.NovaAsyncApi}

        • >> searching params( {osFamily=ubuntu, osVersion=1[012].[01][04], os64Bit=true}

          )

        • << matched image(SYD/23b564c9-c3e6-49f9-bc68-86c7a9ab5018) hardware(SYD/2) location(SYD)
        • >> blocking on sockets [119.9.40.45:22, 10.176.7.31:22] for 600000 MILLISECONDS
        • << socket 119.9.40.45:22 opened
        • <<authenticated>> woke to: net.schmizz.sshj.userauth.UserAuthException: password auth failed
        • << (test12:rsa[fingerprint(03:4f:36:e0:8a:31:e0:7e:85:7a:7c:2b:c9:56:4c:be),sha1(d8:e5:e1:eb:00:af:0e:05:63:61:ee:44:56:4f:5e:96:26:34:3a:df)]@119.9.40.45:22) error acquiring {hostAndPort=119.9.40.45:22, loginUser=test12, ssh=null, connectTimeout=60000, sessionTimeout=60000}

          (attempt 1 of 7): Exhausted available authentication methods

        • <<authenticated>> woke to: net.schmizz.sshj.userauth.UserAuthException: password auth failed
        • << (test12:rsa[fingerprint(03:4f:36:e0:8a:31:e0:7e:85:7a:7c:2b:c9:56:4c:be),sha1(d8:e5:e1:eb:00:af:0e:05:63:61:ee:44:56:4f:5e:96:26:34:3a:df)]@119.9.40.45:22) error acquiring {hostAndPort=119.9.40.45:22, loginUser=test12, ssh=null, connectTimeout=60000, sessionTimeout=60000}

          (attempt 2 of 7): Exhausted available authentication methods

        • <<authenticated>> woke to: net.schmizz.sshj.userauth.UserAuthException: password auth failed
        • << (test12:rsa[fingerprint(03:4f:36:e0:8a:31:e0:7e:85:7a:7c:2b:c9:56:4c:be),sha1(d8:e5:e1:eb:00:af:0e:05:63:61:ee:44:56:4f:5e:96:26:34:3a:df)]@119.9.40.45:22) error acquiring {hostAndPort=119.9.40.45:22, loginUser=test12, ssh=null, connectTimeout=60000, sessionTimeout=60000}

          (attempt 3 of 7): Exhausted available authentication methods

        • <<authenticated>> woke to: net.schmizz.sshj.userauth.UserAuthException: password auth failed
        • << (test12:rsa[fingerprint(03:4f:36:e0:8a:31:e0:7e:85:7a:7c:2b:c9:56:4c:be),sha1(d8:e5:e1:eb:00:af:0e:05:63:61:ee:44:56:4f:5e:96:26:34:3a:df)]@119.9.40.45:22) error acquiring {hostAndPort=119.9.40.45:22, loginUser=test12, ssh=null, connectTimeout=60000, sessionTimeout=60000}

          (attempt 4 of 7): Exhausted available authentication methods

        • <<authenticated>> woke to: net.schmizz.sshj.userauth.UserAuthException: password auth failed
        • << (test12:rsa[fingerprint(03:4f:36:e0:8a:31:e0:7e:85:7a:7c:2b:c9:56:4c:be),sha1(d8:e5:e1:eb:00:af:0e:05:63:61:ee:44:56:4f:5e:96:26:34:3a:df)]@119.9.40.45:22) error acquiring {hostAndPort=119.9.40.45:22, loginUser=test12, ssh=null, connectTimeout=60000, sessionTimeout=60000}

          (attempt 5 of 7): Exhausted available authentication methods

        • <<authenticated>> woke to: net.schmizz.sshj.userauth.UserAuthException: password auth failed
        • << (test12:rsa[fingerprint(03:4f:36:e0:8a:31:e0:7e:85:7a:7c:2b:c9:56:4c:be),sha1(d8:e5:e1:eb:00:af:0e:05:63:61:ee:44:56:4f:5e:96:26:34:3a:df)]@119.9.40.45:22) error acquiring {hostAndPort=119.9.40.45:22, loginUser=test12, ssh=null, connectTimeout=60000, sessionTimeout=60000}

          (attempt 6 of 7): Exhausted available authentication methods

        • <<authenticated>> woke to: net.schmizz.sshj.userauth.UserAuthException: password auth failed
        • << (test12:rsa[fingerprint(03:4f:36:e0:8a:31:e0:7e:85:7a:7c:2b:c9:56:4c:be),sha1(d8:e5:e1:eb:00:af:0e:05:63:61:ee:44:56:4f:5e:96:26:34:3a:df)]@119.9.40.45:22) error acquiring {hostAndPort=119.9.40.45:22, loginUser=test12, ssh=null, connectTimeout=60000, sessionTimeout=60000}

          (out of retries - max 7): Exhausted available authentication methods
          net.schmizz.sshj.userauth.UserAuthException: Exhausted available authentication methods
          at net.schmizz.sshj.userauth.UserAuthImpl.authenticate(UserAuthImpl.java:114) ~[sshj-0.8.1.jar:na]
          at net.schmizz.sshj.SSHClient.auth(SSHClient.java:205) ~[sshj-0.8.1.jar:na]
          at net.schmizz.sshj.SSHClient.auth(SSHClient.java:190) ~[sshj-0.8.1.jar:na]
          at net.schmizz.sshj.SSHClient.authPassword(SSHClient.java:266) ~[sshj-0.8.1.jar:na]
          at net.schmizz.sshj.SSHClient.authPassword(SSHClient.java:236) ~[sshj-0.8.1.jar:na]
          at net.schmizz.sshj.SSHClient.authPassword(SSHClient.java:220) ~[sshj-0.8.1.jar:na]
          at org.jclouds.sshj.SSHClientConnection.create(SSHClientConnection.java:140) ~[jclouds-sshj-1.6.0.jar:1.6.0]
          at org.jclouds.sshj.SSHClientConnection.create(SSHClientConnection.java:40) ~[jclouds-sshj-1.6.0.jar:1.6.0]
          at org.jclouds.sshj.SshjSshClient.acquire(SshjSshClient.java:193) [jclouds-sshj-1.6.0.jar:1.6.0]
          at org.jclouds.sshj.SshjSshClient.connect(SshjSshClient.java:223) [jclouds-sshj-1.6.0.jar:1.6.0]
          at org.jclouds.compute.callables.RunScriptOnNodeUsingSsh.call(RunScriptOnNodeUsingSsh.java:80) ~[jclouds-compute-1.6.0.jar:1.6.0]
          at org.jclouds.compute.strategy.RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:69) ~[jclouds-compute-1.6.0.jar:1.6.0]
          at org.jclouds.compute.strategy.RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:44) ~[jclouds-compute-1.6.0.jar:1.6.0]
          at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) ~[na:1.7.0_25]
          at java.util.concurrent.FutureTask.run(FutureTask.java:166) ~[na:1.7.0_25]
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_25]
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_25]
          at java.lang.Thread.run(Thread.java:724) ~[na:1.7.0_25]
          Caused by: net.schmizz.sshj.userauth.UserAuthException: keyboard-interactive auth not allowed by server
          at net.schmizz.sshj.userauth.UserAuthImpl.authenticate(UserAuthImpl.java:81) ~[sshj-0.8.1.jar:na]
          ... 17 common frames omitted
          << problem applying options to node(SYD/debb4935-a3c4-4e9f-ba49-bc5576dd6545):
          org.jclouds.rest.AuthorizationException: (test12:rsa[fingerprint(03:4f:36:e0:8a:31:e0:7e:85:7a:7c:2b:c9:56:4c:be),sha1(d8:e5:e1:eb:00:af:0e:05:63:61:ee:44:56:4f:5e:96:26:34:3a:df)]@119.9.40.45:22) (test12:rsa[fingerprint(03:4f:36:e0:8a:31:e0:7e:85:7a:7c:2b:c9:56:4c:be),sha1(d8:e5:e1:eb:00:af:0e:05:63:61:ee:44:56:4f:5e:96:26:34:3a:df)]@119.9.40.45:22) error acquiring

          {hostAndPort=119.9.40.45:22, loginUser=test12, ssh=null, connectTimeout=60000, sessionTimeout=60000}

          (out of retries - max 7): Exhausted available authentication methods
          at org.jclouds.sshj.SshjSshClient.propagate(SshjSshClient.java:382)
          at org.jclouds.sshj.SshjSshClient.acquire(SshjSshClient.java:203)
          at org.jclouds.sshj.SshjSshClient.connect(SshjSshClient.java:223)
          at org.jclouds.compute.callables.RunScriptOnNodeUsingSsh.call(RunScriptOnNodeUsingSsh.java:80)
          at org.jclouds.compute.strategy.RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:69)
          at org.jclouds.compute.strategy.RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:44)
          at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
          at java.util.concurrent.FutureTask.run(FutureTask.java:166)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          at java.lang.Thread.run(Thread.java:724)
          Caused by: net.schmizz.sshj.userauth.UserAuthException: Exhausted available authentication methods
          at net.schmizz.sshj.userauth.UserAuthImpl.authenticate(UserAuthImpl.java:114)
          at net.schmizz.sshj.SSHClient.auth(SSHClient.java:205)
          at net.schmizz.sshj.SSHClient.auth(SSHClient.java:190)
          at net.schmizz.sshj.SSHClient.authPassword(SSHClient.java:266)
          at net.schmizz.sshj.SSHClient.authPassword(SSHClient.java:236)
          at net.schmizz.sshj.SSHClient.authPassword(SSHClient.java:220)
          at org.jclouds.sshj.SSHClientConnection.create(SSHClientConnection.java:140)
          at org.jclouds.sshj.SSHClientConnection.create(SSHClientConnection.java:40)
          at org.jclouds.sshj.SshjSshClient.acquire(SshjSshClient.java:193)
          ... 9 more
          Caused by: net.schmizz.sshj.userauth.UserAuthException: keyboard-interactive auth not allowed by server
          at net.schmizz.sshj.userauth.UserAuthImpl.authenticate(UserAuthImpl.java:81)
          ... 17 more
          error installing apt,apache2,test_cb,networking_basic on group testnode128: error runScript on filtered nodes options(

          {loginUser=test12, loginPasswordPresent=true, loginPrivateKeyPresent=true, runAsRoot=false, wrapInInitScript=false}

          )
          Execution failures:

        0 error[s]
        Node failures:

        1) AuthorizationException on node SYD/debb4935-a3c4-4e9f-ba49-bc5576dd6545:
        org.jclouds.rest.AuthorizationException: (test12:rsa[fingerprint(03:4f:36:e0:8a:31:e0:7e:85:7a:7c:2b:c9:56:4c:be),sha1(d8:e5:e1:eb:00:af:0e:05:63:61:ee:44:56:4f:5e:96:26:34:3a:df)]@119.9.40.45:22) (test12:rsa[fingerprint(03:4f:36:e0:8a:31:e0:7e:85:7a:7c:2b:c9:56:4c:be),sha1(d8:e5:e1:eb:00:af:0e:05:63:61:ee:44:56:4f:5e:96:26:34:3a:df)]@119.9.40.45:22) error acquiring

        {hostAndPort=119.9.40.45:22, loginUser=test12, ssh=null, connectTimeout=60000, sessionTimeout=60000}

        (out of retries - max 7): Exhausted available authentication methods
        at org.jclouds.sshj.SshjSshClient.propagate(SshjSshClient.java:382)
        at org.jclouds.sshj.SshjSshClient.acquire(SshjSshClient.java:203)
        at org.jclouds.sshj.SshjSshClient.connect(SshjSshClient.java:223)
        at org.jclouds.compute.callables.RunScriptOnNodeUsingSsh.call(RunScriptOnNodeUsingSsh.java:80)
        at org.jclouds.compute.strategy.RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:69)
        at org.jclouds.compute.strategy.RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:44)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:724)
        Caused by: net.schmizz.sshj.userauth.UserAuthException: Exhausted available authentication methods
        at net.schmizz.sshj.userauth.UserAuthImpl.authenticate(UserAuthImpl.java:114)
        at net.schmizz.sshj.SSHClient.auth(SSHClient.java:205)
        at net.schmizz.sshj.SSHClient.auth(SSHClient.java:190)
        at net.schmizz.sshj.SSHClient.authPassword(SSHClient.java:266)
        at net.schmizz.sshj.SSHClient.authPassword(SSHClient.java:236)
        at net.schmizz.sshj.SSHClient.authPassword(SSHClient.java:220)
        at org.jclouds.sshj.SSHClientConnection.create(SSHClientConnection.java:140)
        at org.jclouds.sshj.SSHClientConnection.create(SSHClientConnection.java:40)
        at org.jclouds.sshj.SshjSshClient.acquire(SshjSshClient.java:193)
        ... 9 more
        Caused by: net.schmizz.sshj.userauth.UserAuthException: keyboard-interactive auth not allowed by server
        at net.schmizz.sshj.userauth.UserAuthImpl.authenticate(UserAuthImpl.java:81)
        ... 17 more

        Show
        nareshparimi added a comment - - edited I have solved this issue:--- I have stuck with this issue and googled a lot for hints ,But finally i have solved this issue myself . Hope this post is useful. Root cause for this problem : The non existence of the user, we are trying to runscripts on VM side. Solution: Make sure the user we are using is existing and check whether we are able to do SSH with that user onto VM machine. ERROR LOG: initializing {id=openstack-nova, name=OpenStack Nova Diablo+ API, views= [org.jclouds.compute.ComputeServiceContext] , endpointName=KeyStone base url ending in /v2.0/, identityName=$ {tenantName} :$ {userName} or ${userName} , if your keystone supports a default tenant, credentialName=Optional.of($ {password} ), documentation= http://api.openstack.org/ , api=interface org.jclouds.openstack.nova.v2_0.NovaApi, asyncApi=interface org.jclouds.openstack.nova.v2_0.NovaAsyncApi} >> searching params( {osFamily=ubuntu, osVersion=1[012].[01][04], os64Bit=true} ) << matched image(SYD/23b564c9-c3e6-49f9-bc68-86c7a9ab5018) hardware(SYD/2) location(SYD) >> blocking on sockets [119.9.40.45:22, 10.176.7.31:22] for 600000 MILLISECONDS << socket 119.9.40.45:22 opened <<authenticated>> woke to: net.schmizz.sshj.userauth.UserAuthException: password auth failed << (test12:rsa [fingerprint(03:4f:36:e0:8a:31:e0:7e:85:7a:7c:2b:c9:56:4c:be),sha1(d8:e5:e1:eb:00:af:0e:05:63:61:ee:44:56:4f:5e:96:26:34:3a:df)] @119.9.40.45:22) error acquiring {hostAndPort=119.9.40.45:22, loginUser=test12, ssh=null, connectTimeout=60000, sessionTimeout=60000} (attempt 1 of 7): Exhausted available authentication methods <<authenticated>> woke to: net.schmizz.sshj.userauth.UserAuthException: password auth failed << (test12:rsa [fingerprint(03:4f:36:e0:8a:31:e0:7e:85:7a:7c:2b:c9:56:4c:be),sha1(d8:e5:e1:eb:00:af:0e:05:63:61:ee:44:56:4f:5e:96:26:34:3a:df)] @119.9.40.45:22) error acquiring {hostAndPort=119.9.40.45:22, loginUser=test12, ssh=null, connectTimeout=60000, sessionTimeout=60000} (attempt 2 of 7): Exhausted available authentication methods <<authenticated>> woke to: net.schmizz.sshj.userauth.UserAuthException: password auth failed << (test12:rsa [fingerprint(03:4f:36:e0:8a:31:e0:7e:85:7a:7c:2b:c9:56:4c:be),sha1(d8:e5:e1:eb:00:af:0e:05:63:61:ee:44:56:4f:5e:96:26:34:3a:df)] @119.9.40.45:22) error acquiring {hostAndPort=119.9.40.45:22, loginUser=test12, ssh=null, connectTimeout=60000, sessionTimeout=60000} (attempt 3 of 7): Exhausted available authentication methods <<authenticated>> woke to: net.schmizz.sshj.userauth.UserAuthException: password auth failed << (test12:rsa [fingerprint(03:4f:36:e0:8a:31:e0:7e:85:7a:7c:2b:c9:56:4c:be),sha1(d8:e5:e1:eb:00:af:0e:05:63:61:ee:44:56:4f:5e:96:26:34:3a:df)] @119.9.40.45:22) error acquiring {hostAndPort=119.9.40.45:22, loginUser=test12, ssh=null, connectTimeout=60000, sessionTimeout=60000} (attempt 4 of 7): Exhausted available authentication methods <<authenticated>> woke to: net.schmizz.sshj.userauth.UserAuthException: password auth failed << (test12:rsa [fingerprint(03:4f:36:e0:8a:31:e0:7e:85:7a:7c:2b:c9:56:4c:be),sha1(d8:e5:e1:eb:00:af:0e:05:63:61:ee:44:56:4f:5e:96:26:34:3a:df)] @119.9.40.45:22) error acquiring {hostAndPort=119.9.40.45:22, loginUser=test12, ssh=null, connectTimeout=60000, sessionTimeout=60000} (attempt 5 of 7): Exhausted available authentication methods <<authenticated>> woke to: net.schmizz.sshj.userauth.UserAuthException: password auth failed << (test12:rsa [fingerprint(03:4f:36:e0:8a:31:e0:7e:85:7a:7c:2b:c9:56:4c:be),sha1(d8:e5:e1:eb:00:af:0e:05:63:61:ee:44:56:4f:5e:96:26:34:3a:df)] @119.9.40.45:22) error acquiring {hostAndPort=119.9.40.45:22, loginUser=test12, ssh=null, connectTimeout=60000, sessionTimeout=60000} (attempt 6 of 7): Exhausted available authentication methods <<authenticated>> woke to: net.schmizz.sshj.userauth.UserAuthException: password auth failed << (test12:rsa [fingerprint(03:4f:36:e0:8a:31:e0:7e:85:7a:7c:2b:c9:56:4c:be),sha1(d8:e5:e1:eb:00:af:0e:05:63:61:ee:44:56:4f:5e:96:26:34:3a:df)] @119.9.40.45:22) error acquiring {hostAndPort=119.9.40.45:22, loginUser=test12, ssh=null, connectTimeout=60000, sessionTimeout=60000} (out of retries - max 7): Exhausted available authentication methods net.schmizz.sshj.userauth.UserAuthException: Exhausted available authentication methods at net.schmizz.sshj.userauth.UserAuthImpl.authenticate(UserAuthImpl.java:114) ~ [sshj-0.8.1.jar:na] at net.schmizz.sshj.SSHClient.auth(SSHClient.java:205) ~ [sshj-0.8.1.jar:na] at net.schmizz.sshj.SSHClient.auth(SSHClient.java:190) ~ [sshj-0.8.1.jar:na] at net.schmizz.sshj.SSHClient.authPassword(SSHClient.java:266) ~ [sshj-0.8.1.jar:na] at net.schmizz.sshj.SSHClient.authPassword(SSHClient.java:236) ~ [sshj-0.8.1.jar:na] at net.schmizz.sshj.SSHClient.authPassword(SSHClient.java:220) ~ [sshj-0.8.1.jar:na] at org.jclouds.sshj.SSHClientConnection.create(SSHClientConnection.java:140) ~ [jclouds-sshj-1.6.0.jar:1.6.0] at org.jclouds.sshj.SSHClientConnection.create(SSHClientConnection.java:40) ~ [jclouds-sshj-1.6.0.jar:1.6.0] at org.jclouds.sshj.SshjSshClient.acquire(SshjSshClient.java:193) [jclouds-sshj-1.6.0.jar:1.6.0] at org.jclouds.sshj.SshjSshClient.connect(SshjSshClient.java:223) [jclouds-sshj-1.6.0.jar:1.6.0] at org.jclouds.compute.callables.RunScriptOnNodeUsingSsh.call(RunScriptOnNodeUsingSsh.java:80) ~ [jclouds-compute-1.6.0.jar:1.6.0] at org.jclouds.compute.strategy.RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:69) ~ [jclouds-compute-1.6.0.jar:1.6.0] at org.jclouds.compute.strategy.RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:44) ~ [jclouds-compute-1.6.0.jar:1.6.0] at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) ~ [na:1.7.0_25] at java.util.concurrent.FutureTask.run(FutureTask.java:166) ~ [na:1.7.0_25] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~ [na:1.7.0_25] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~ [na:1.7.0_25] at java.lang.Thread.run(Thread.java:724) ~ [na:1.7.0_25] Caused by: net.schmizz.sshj.userauth.UserAuthException: keyboard-interactive auth not allowed by server at net.schmizz.sshj.userauth.UserAuthImpl.authenticate(UserAuthImpl.java:81) ~ [sshj-0.8.1.jar:na] ... 17 common frames omitted << problem applying options to node(SYD/debb4935-a3c4-4e9f-ba49-bc5576dd6545): org.jclouds.rest.AuthorizationException: (test12:rsa [fingerprint(03:4f:36:e0:8a:31:e0:7e:85:7a:7c:2b:c9:56:4c:be),sha1(d8:e5:e1:eb:00:af:0e:05:63:61:ee:44:56:4f:5e:96:26:34:3a:df)] @119.9.40.45:22) (test12:rsa [fingerprint(03:4f:36:e0:8a:31:e0:7e:85:7a:7c:2b:c9:56:4c:be),sha1(d8:e5:e1:eb:00:af:0e:05:63:61:ee:44:56:4f:5e:96:26:34:3a:df)] @119.9.40.45:22) error acquiring {hostAndPort=119.9.40.45:22, loginUser=test12, ssh=null, connectTimeout=60000, sessionTimeout=60000} (out of retries - max 7): Exhausted available authentication methods at org.jclouds.sshj.SshjSshClient.propagate(SshjSshClient.java:382) at org.jclouds.sshj.SshjSshClient.acquire(SshjSshClient.java:203) at org.jclouds.sshj.SshjSshClient.connect(SshjSshClient.java:223) at org.jclouds.compute.callables.RunScriptOnNodeUsingSsh.call(RunScriptOnNodeUsingSsh.java:80) at org.jclouds.compute.strategy.RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:69) at org.jclouds.compute.strategy.RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:44) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724) Caused by: net.schmizz.sshj.userauth.UserAuthException: Exhausted available authentication methods at net.schmizz.sshj.userauth.UserAuthImpl.authenticate(UserAuthImpl.java:114) at net.schmizz.sshj.SSHClient.auth(SSHClient.java:205) at net.schmizz.sshj.SSHClient.auth(SSHClient.java:190) at net.schmizz.sshj.SSHClient.authPassword(SSHClient.java:266) at net.schmizz.sshj.SSHClient.authPassword(SSHClient.java:236) at net.schmizz.sshj.SSHClient.authPassword(SSHClient.java:220) at org.jclouds.sshj.SSHClientConnection.create(SSHClientConnection.java:140) at org.jclouds.sshj.SSHClientConnection.create(SSHClientConnection.java:40) at org.jclouds.sshj.SshjSshClient.acquire(SshjSshClient.java:193) ... 9 more Caused by: net.schmizz.sshj.userauth.UserAuthException: keyboard-interactive auth not allowed by server at net.schmizz.sshj.userauth.UserAuthImpl.authenticate(UserAuthImpl.java:81) ... 17 more error installing apt,apache2,test_cb,networking_basic on group testnode128: error runScript on filtered nodes options( {loginUser=test12, loginPasswordPresent=true, loginPrivateKeyPresent=true, runAsRoot=false, wrapInInitScript=false} ) Execution failures: 0 error [s] Node failures: 1) AuthorizationException on node SYD/debb4935-a3c4-4e9f-ba49-bc5576dd6545: org.jclouds.rest.AuthorizationException: (test12:rsa [fingerprint(03:4f:36:e0:8a:31:e0:7e:85:7a:7c:2b:c9:56:4c:be),sha1(d8:e5:e1:eb:00:af:0e:05:63:61:ee:44:56:4f:5e:96:26:34:3a:df)] @119.9.40.45:22) (test12:rsa [fingerprint(03:4f:36:e0:8a:31:e0:7e:85:7a:7c:2b:c9:56:4c:be),sha1(d8:e5:e1:eb:00:af:0e:05:63:61:ee:44:56:4f:5e:96:26:34:3a:df)] @119.9.40.45:22) error acquiring {hostAndPort=119.9.40.45:22, loginUser=test12, ssh=null, connectTimeout=60000, sessionTimeout=60000} (out of retries - max 7): Exhausted available authentication methods at org.jclouds.sshj.SshjSshClient.propagate(SshjSshClient.java:382) at org.jclouds.sshj.SshjSshClient.acquire(SshjSshClient.java:203) at org.jclouds.sshj.SshjSshClient.connect(SshjSshClient.java:223) at org.jclouds.compute.callables.RunScriptOnNodeUsingSsh.call(RunScriptOnNodeUsingSsh.java:80) at org.jclouds.compute.strategy.RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:69) at org.jclouds.compute.strategy.RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:44) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724) Caused by: net.schmizz.sshj.userauth.UserAuthException: Exhausted available authentication methods at net.schmizz.sshj.userauth.UserAuthImpl.authenticate(UserAuthImpl.java:114) at net.schmizz.sshj.SSHClient.auth(SSHClient.java:205) at net.schmizz.sshj.SSHClient.auth(SSHClient.java:190) at net.schmizz.sshj.SSHClient.authPassword(SSHClient.java:266) at net.schmizz.sshj.SSHClient.authPassword(SSHClient.java:236) at net.schmizz.sshj.SSHClient.authPassword(SSHClient.java:220) at org.jclouds.sshj.SSHClientConnection.create(SSHClientConnection.java:140) at org.jclouds.sshj.SSHClientConnection.create(SSHClientConnection.java:40) at org.jclouds.sshj.SshjSshClient.acquire(SshjSshClient.java:193) ... 9 more Caused by: net.schmizz.sshj.userauth.UserAuthException: keyboard-interactive auth not allowed by server at net.schmizz.sshj.userauth.UserAuthImpl.authenticate(UserAuthImpl.java:81) ... 17 more

          People

          • Assignee:
            Adrian Cole
            Reporter:
            Marc de Palol
          • Votes:
            4 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development