Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.10.0
    • Component/s: camel-ssh
    • Labels:
      None
    • Estimated Complexity:
      Unknown

      Description

      Lately, I've seen a lot of questions in the mailing list for people that would want to use camel to execute SSH commands to remote hosts.
      I think it would be nice if we could provide such component to the users.

      1. camel-ssh-certFilename.patch
        11 kB
        Scott Cranton
      2. camel-ssh.patch
        79 kB
        Scott Cranton
      3. camel-ssh-patch.diff
        11 kB
        Scott Cranton
      4. README.md
        1 kB
        Scott Cranton

        Activity

        Hide
        Hadrian Zbarcea added a comment -

        The camel-exec component already offers this feature. The only difference would be that camel-exec won't keep a connection to the remote server open.

        Show
        Hadrian Zbarcea added a comment - The camel-exec component already offers this feature. The only difference would be that camel-exec won't keep a connection to the remote server open.
        Hide
        Heath Kesler added a comment -

        is there a need for extended open connections? Maybe a parameter for this component could be provided to allow the user to specify maintaining the connection after executing a command?

        Show
        Heath Kesler added a comment - is there a need for extended open connections? Maybe a parameter for this component could be provided to allow the user to specify maintaining the connection after executing a command?
        Hide
        Jason Whaley added a comment -

        I've got some preliminary code that at least covers the basics out on github right now. It is lacking some parameterization (such as what Heath mentioned).

        https://github.com/whaley/camel-ssh

        Right now with the code as it is written you need to hit another server (or localhost) running ssh but I'm soon to embed mina-sshd for the tests.

        Show
        Jason Whaley added a comment - I've got some preliminary code that at least covers the basics out on github right now. It is lacking some parameterization (such as what Heath mentioned). https://github.com/whaley/camel-ssh Right now with the code as it is written you need to hit another server (or localhost) running ssh but I'm soon to embed mina-sshd for the tests.
        Hide
        Jason Whaley added a comment -

        I've added some parameterization to this and it is working, at least according to some test methods I've written that embed apache sshd (warning: they aren't fast tests). Right now it supports retry logic, keeps a connection alive across multiple message exchanges, and allows you to use key or password based auth (though right now it expects a private key file from a File reference instead of from class path or from a keystore).

        Could someone give it a look and provide some feedback on this code as it exists?

        Show
        Jason Whaley added a comment - I've added some parameterization to this and it is working, at least according to some test methods I've written that embed apache sshd (warning: they aren't fast tests). Right now it supports retry logic, keeps a connection alive across multiple message exchanges, and allows you to use key or password based auth (though right now it expects a private key file from a File reference instead of from class path or from a keystore). Could someone give it a look and provide some feedback on this code as it exists?
        Hide
        Scott Cranton added a comment -

        Looks like a couple of us have the same thought. I've been working on a Mina SSHD client based component as well

        https://github.com/scranton/camel/tree/camel-ssh

        Mine uses the following syntax "ssh://[user[:password]@]host[:port][?queryParams]", and I'm using the exchange body as the command(s) to send to the SSH Server. I was thinking of also allowing this to be a ScheduledPollConsumer using a "pollCommand=<command>" parameter to avoid the timer;setBody;to uri="ssh" boilerplate.

        I've got it merged in the camel trunk with a Mina SSHD unit test, an example in examples/camel-example-ssh, and I've added some OSGi stuff such that you can deploy the example in Karaf 2.2.4 and at least the simple example works...

        Jason: would love to work with you on this. Let me take a look at what you've already done, and let's see how we can collaborate...

        Show
        Scott Cranton added a comment - Looks like a couple of us have the same thought. I've been working on a Mina SSHD client based component as well https://github.com/scranton/camel/tree/camel-ssh Mine uses the following syntax "ssh://[user [:password] @]host [:port] [?queryParams] ", and I'm using the exchange body as the command(s) to send to the SSH Server. I was thinking of also allowing this to be a ScheduledPollConsumer using a "pollCommand=<command>" parameter to avoid the timer;setBody;to uri="ssh" boilerplate. I've got it merged in the camel trunk with a Mina SSHD unit test, an example in examples/camel-example-ssh, and I've added some OSGi stuff such that you can deploy the example in Karaf 2.2.4 and at least the simple example works... Jason: would love to work with you on this. Let me take a look at what you've already done, and let's see how we can collaborate...
        Hide
        Jason Whaley added a comment -

        Sure! The route you took using the client stuff in apache sshd was probably wiser, since I've got a fair bit of protocol level code using jsch that's a bit of an eyesore and maintenance burden. Just so long as we can do something like support streaming (which I was going to get to eventually) and re-use of a single connection per producer sshd's client classes ought to be ok. I wasn't near finished with all I wanted to do, but I've got a fleshed out set of tests that were testing things like retries and sending multiple commands.

        I'm around on the camel irc channel on codehaus if you want to sync up there soon. Let's figure out what should be in the component and nail down a final URL structure (mine differs a fair bit from yours).

        Show
        Jason Whaley added a comment - Sure! The route you took using the client stuff in apache sshd was probably wiser, since I've got a fair bit of protocol level code using jsch that's a bit of an eyesore and maintenance burden. Just so long as we can do something like support streaming (which I was going to get to eventually) and re-use of a single connection per producer sshd's client classes ought to be ok. I wasn't near finished with all I wanted to do, but I've got a fleshed out set of tests that were testing things like retries and sending multiple commands. I'm around on the camel irc channel on codehaus if you want to sync up there soon. Let's figure out what should be in the component and nail down a final URL structure (mine differs a fair bit from yours).
        Hide
        Ioannis Canellos added a comment -

        Some comments about the ssh clients in java. So far I know of 3 efforts.
        i) Jsch
        ii) Mina SSH
        iii) sshj https://github.com/shikhar/sshj

        From personal experience jsch is widely used, but in some cases it has its issues (for example there are cases where it needs quite a few retries with no apparent reason). I was using in jclouds-karaf for running scripts to remote hosts and ditched it.

        I am a huge fan of mina, but I am not sure if its ssh client is of equivalent quality with the server. I haven't seen projects using it and I have not used it myself.

        SSHJ works like a charm, is OSGi ready, supports all the things we would want from an ssh client. There are a couple of projects using it and my personal experience with is has been great.

        So my opinion is:

        -1 jsch
        +0 mina (if it works without issues and supports things like rsa key auth etc, I would go +1, but have no personal experience).
        +1 sshj

        Show
        Ioannis Canellos added a comment - Some comments about the ssh clients in java. So far I know of 3 efforts. i) Jsch ii) Mina SSH iii) sshj https://github.com/shikhar/sshj From personal experience jsch is widely used, but in some cases it has its issues (for example there are cases where it needs quite a few retries with no apparent reason). I was using in jclouds-karaf for running scripts to remote hosts and ditched it. I am a huge fan of mina, but I am not sure if its ssh client is of equivalent quality with the server. I haven't seen projects using it and I have not used it myself. SSHJ works like a charm, is OSGi ready, supports all the things we would want from an ssh client. There are a couple of projects using it and my personal experience with is has been great. So my opinion is: -1 jsch +0 mina (if it works without issues and supports things like rsa key auth etc, I would go +1, but have no personal experience). +1 sshj
        Hide
        Scott Cranton added a comment - - edited

        I'm using Mina SSH client in my flavor (https://github.com/scranton/camel), and no issues with it so far...

        I've got Polling Consumer and Producer working, a simple example project, mods to the OSGi feature stuff so it deploys into at least Karaf 2.2.4 (my simple example deploys and runs in Karaf calling Karaf via SSH). I'm planning on adding some re-connect logic, and rsa key auth over next day or two (I'll look at some of Jason's test cases to pull over).

        Any testers / feedback / help greatly appreciated...

        Show
        Scott Cranton added a comment - - edited I'm using Mina SSH client in my flavor ( https://github.com/scranton/camel ), and no issues with it so far... I've got Polling Consumer and Producer working, a simple example project, mods to the OSGi feature stuff so it deploys into at least Karaf 2.2.4 (my simple example deploys and runs in Karaf calling Karaf via SSH). I'm planning on adding some re-connect logic, and rsa key auth over next day or two (I'll look at some of Jason's test cases to pull over). Any testers / feedback / help greatly appreciated...
        Hide
        Ioannis Canellos added a comment -

        Its good to hear that mina ssh client is up to the task. I'll gladly try it out and provide feedback or any other help you'd like.

        Show
        Ioannis Canellos added a comment - Its good to hear that mina ssh client is up to the task. I'll gladly try it out and provide feedback or any other help you'd like.
        Hide
        Scott Cranton added a comment -

        I've now got a connection re-connect test, and code and test for RSA Key Auth at https://github.com/scranton/camel. Plus sync'd with Camel trunk as of today...

        Any feedback, testing, etc. would be greatly appreciated...

        Not sure at one point this component reaches enough critical mass to be worthy of consideration of inclusion in Camel proper for on-going use and development...

        Show
        Scott Cranton added a comment - I've now got a connection re-connect test, and code and test for RSA Key Auth at https://github.com/scranton/camel . Plus sync'd with Camel trunk as of today... Any feedback, testing, etc. would be greatly appreciated... Not sure at one point this component reaches enough critical mass to be worthy of consideration of inclusion in Camel proper for on-going use and development...
        Hide
        Claus Ibsen added a comment -

        Good work Scott.

        In the endpoint, I would suggest not to both log error and throw exception. Just throwing exception is fine. And use CamelExechangeException instead of just Exception.

        And add some debug/trace logging in the sendExecCommand so ppl can turn on logging to see what the component is doing.

        Does the await() method block forever? What would happen if there is no physical connection? I assume it should be possible to set some timeout value.

        In the producer, you may be aware that getOut will create a new empty message. Which means that any headers is not preserved. So what we usually do is to copy those headers. See for example TransformProcessor from camel-core.

        Show
        Claus Ibsen added a comment - Good work Scott. In the endpoint, I would suggest not to both log error and throw exception. Just throwing exception is fine. And use CamelExechangeException instead of just Exception. And add some debug/trace logging in the sendExecCommand so ppl can turn on logging to see what the component is doing. Does the await() method block forever? What would happen if there is no physical connection? I assume it should be possible to set some timeout value. In the producer, you may be aware that getOut will create a new empty message. Which means that any headers is not preserved. So what we usually do is to copy those headers. See for example TransformProcessor from camel-core.
        Hide
        Scott Cranton added a comment -

        Claus: Thanks for the feedback - all your suggestions have been incorporated...

        Show
        Scott Cranton added a comment - Claus: Thanks for the feedback - all your suggestions have been incorporated...
        Hide
        Claus Ibsen added a comment -

        Scott this is great. I guess make sure you have adequate unit test coverage as well, if not already.
        Also you may polish the javadoc a bit and remove the Readme.txt file which is from the maven archetype etc.

        As well give some thoughts of the documentation for this new component.

        Jason, you got any feedback / thoughts?

        Show
        Claus Ibsen added a comment - Scott this is great. I guess make sure you have adequate unit test coverage as well, if not already. Also you may polish the javadoc a bit and remove the Readme.txt file which is from the maven archetype etc. As well give some thoughts of the documentation for this new component. Jason, you got any feedback / thoughts?
        Hide
        Jason Whaley added a comment -

        I just sent in a pull request that includes retry support in case you can't connect initially - there are two extra configuration parameters I've added, maximumReconnectAttempts and reconnectDelay.

        Also, does each exchange that gets sent to this endpoint use the same connection when using SshClient from sshd-core? That isn't immediately obvious to me on first glance at SshClient.

        Show
        Jason Whaley added a comment - I just sent in a pull request that includes retry support in case you can't connect initially - there are two extra configuration parameters I've added, maximumReconnectAttempts and reconnectDelay. Also, does each exchange that gets sent to this endpoint use the same connection when using SshClient from sshd-core? That isn't immediately obvious to me on first glance at SshClient.
        Hide
        Scott Cranton added a comment -

        Jason,

        I added a unit test using Camel Error Handler redeliver, and it provides equivalent / better capabilities than your maxReconnectAttempts code, so I'm reverting your code for now. Please let me know if you agree...

        I need to dig deeper into Mina SSH client code to understand if it holds the connection open across requests. Knowing that may cause me to re-think adding in reconnect logic within the camel-ssh component.

        Thanks,
        Scott

        Show
        Scott Cranton added a comment - Jason, I added a unit test using Camel Error Handler redeliver, and it provides equivalent / better capabilities than your maxReconnectAttempts code, so I'm reverting your code for now. Please let me know if you agree... I need to dig deeper into Mina SSH client code to understand if it holds the connection open across requests. Knowing that may cause me to re-think adding in reconnect logic within the camel-ssh component. Thanks, Scott
        Hide
        Jason Whaley added a comment -

        Pending research in to how the ssh client code handles reusing the connection.. having redelivery set via configuration would allow the redelivery to be controlled per endpoint instead of per route, but I suppose that can be worked around pretty easily if the route is kept short.

        I don't have a strong opinion either way as to whether or not we do it via configuration of the endpoint or in an error handler if there's no real impact from whether the same connection is reused across multiple exchanges.

        Show
        Jason Whaley added a comment - Pending research in to how the ssh client code handles reusing the connection.. having redelivery set via configuration would allow the redelivery to be controlled per endpoint instead of per route, but I suppose that can be worked around pretty easily if the route is kept short. I don't have a strong opinion either way as to whether or not we do it via configuration of the endpoint or in an error handler if there's no real impact from whether the same connection is reused across multiple exchanges.
        Hide
        Ioannis Canellos added a comment -

        Great job Scott.

        I haven't had the chance to test the component, but I have given a glimpse at the source.
        Assuming that Mina SSH clients works in the same manner as Mina I assume that the client connection will stay open across requests. With this in mind it might be a good idea to hold the session open[1] and reuse it across requests, so that you don't connect, authenticate etc all the time.

        Hopefully, I'll find some more time and provide more feedback.

        [1]: There could be an enpoint uri parameter or a header in the Exchange that would allow the user that doesn't want the connection to be open. We have something like this in camel-mina, camel-mail etc.

        Show
        Ioannis Canellos added a comment - Great job Scott. I haven't had the chance to test the component, but I have given a glimpse at the source. Assuming that Mina SSH clients works in the same manner as Mina I assume that the client connection will stay open across requests. With this in mind it might be a good idea to hold the session open [1] and reuse it across requests, so that you don't connect, authenticate etc all the time. Hopefully, I'll find some more time and provide more feedback. [1] : There could be an enpoint uri parameter or a header in the Exchange that would allow the user that doesn't want the connection to be open. We have something like this in camel-mina, camel-mail etc.
        Hide
        Scott Cranton added a comment -

        Here's a patch file including the camel-ssh component with example, unit tests, and OSGi feature mods...

        Show
        Scott Cranton added a comment - Here's a patch file including the camel-ssh component with example, unit tests, and OSGi feature mods...
        Hide
        Claus Ibsen added a comment -

        Scott, there is something wrong with the patch. It does not contain the full source of the camel-ssh component.

        Show
        Claus Ibsen added a comment - Scott, there is something wrong with the patch. It does not contain the full source of the camel-ssh component.
        Hide
        Scott Cranton added a comment -

        I created this patch file using diff... looks like there are errors in other patch file in the components/pom.xml such that it won't merge in adding the <module>camel-ssh</module> change...

        Let me know if this new patch works better - thanks

        Show
        Scott Cranton added a comment - I created this patch file using diff... looks like there are errors in other patch file in the components/pom.xml such that it won't merge in adding the <module>camel-ssh</module> change... Let me know if this new patch works better - thanks
        Hide
        Scott Cranton added a comment -

        I also created a GitHub pull request - https://github.com/apache/camel/pull/2 - for this...

        Sorry I'm still a newbie with git...

        Show
        Scott Cranton added a comment - I also created a GitHub pull request - https://github.com/apache/camel/pull/2 - for this... Sorry I'm still a newbie with git...
        Hide
        Claus Ibsen added a comment -

        Scott, Jason, thanks for your contribution. Its added to ASF now.

        I put up an initial wiki page for the documentation at
        https://cwiki.apache.org/confluence/display/CAMEL/SSH

        There is most likely a few options which is not documented in the table. And we need some examples as well.
        Can you help with that?

        Show
        Claus Ibsen added a comment - Scott, Jason, thanks for your contribution. Its added to ASF now. I put up an initial wiki page for the documentation at https://cwiki.apache.org/confluence/display/CAMEL/SSH There is most likely a few options which is not documented in the table. And we need some examples as well. Can you help with that?
        Hide
        Claus Ibsen added a comment -

        Updated the documentation a bit.

        I suggest that we provide an option for more easily use the KeyPairProvider with a file, so you can just refer to the file name, and Camel will automatic create and use a FileKeyPairProvider. This makes configuration from XML DSL much easier.

        Show
        Claus Ibsen added a comment - Updated the documentation a bit. I suggest that we provide an option for more easily use the KeyPairProvider with a file, so you can just refer to the file name, and Camel will automatic create and use a FileKeyPairProvider. This makes configuration from XML DSL much easier.
        Hide
        Scott Cranton added a comment -

        Claus,

        Agreed. I'll try to today / tomorrow get a new option for the cert file using FileKeyPairProvider (I'll also look at classpath option), and adding to the camel-example-ssh an XML setup for Cert.

        Thanks,
        Scott

        Show
        Scott Cranton added a comment - Claus, Agreed. I'll try to today / tomorrow get a new option for the cert file using FileKeyPairProvider (I'll also look at classpath option), and adding to the camel-example-ssh an XML setup for Cert. Thanks, Scott
        Hide
        Scott Cranton added a comment -

        Patch adding certFilename parameter so that you can just specify a certificate filename in the uri.

        ssh://smx@localhost:22?certFilename=foo.pem

        Show
        Scott Cranton added a comment - Patch adding certFilename parameter so that you can just specify a certificate filename in the uri. ssh://smx@localhost:22?certFilename=foo.pem
        Hide
        Claus Ibsen added a comment -

        Thanks for the patch Scott. I have applied it to trunk, and updated the wiki page.

        Show
        Claus Ibsen added a comment - Thanks for the patch Scott. I have applied it to trunk, and updated the wiki page.

          People

          • Assignee:
            Claus Ibsen
            Reporter:
            Ioannis Canellos
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development