Camel
  1. Camel
  2. CAMEL-3476

Contribute camel-sns component to the new camel-aws component.

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 2.6.0
    • Fix Version/s: 2.8.0
    • Component/s: camel-aws
    • Labels:
      None

      Description

      CAMEL-3468 contributes a new component for interacting with the Amazon SQS service. The camel-sns component allows users to publish/subscribe to topics on Amazon's Simple Notification Service.

      This code is currently hosted as a google code project here:

      http://code.google.com/p/camel-sns/

      I'll coordinate with Tracy (developer of camel-aws) in the creation of the patch.

      1. camel-aws-3476.txt
        89 kB
        Mark Ford
      2. patchfile.txt
        114 kB
        Tracy Snell
      3. patchfile.txt
        113 kB
        Tracy Snell

        Activity

        Hide
        Tracy Snell added a comment -

        I've repackaged the code in prep for Mark to submit it but waiting on Amazon to figure out my SNS access issue to confirm tests.

        Show
        Tracy Snell added a comment - I've repackaged the code in prep for Mark to submit it but waiting on Amazon to figure out my SNS access issue to confirm tests.
        Hide
        Tracy Snell added a comment -

        Code repacked. Some minor tweaks. Ready for Mark to submit.

        Show
        Tracy Snell added a comment - Code repacked. Some minor tweaks. Ready for Mark to submit.
        Hide
        Mark Ford added a comment -

        Attached is my camel-sns contribution that has been repackaged by Tracy Snell. This code should be merged with the camel-aws contribution in order to leverage code between the SQS and SNS components.

        Note that some of the integration tests with SNS are slow due to the delay involved in propagating the permissions for the target queue. At one point the advertised propagation time was 60 seconds but in practice it was over 90 seconds. All of the integration tests that rely on this propagation wait 120 seconds to be sure. A better approach would be to setup a single queue for the integration tests and reuse it for all of the tests. This is left for a future submission (i.e. from Tracy).

        Additionally, some of the suggestions for the SQS component submission apply here as well. For example, the setting of the Amazon API object, public key, etc would be nice improvements.

        There are a few FIXME's left undone but they don't affect the functionality. One is a minor tweak and two are missing tests. I'm submitting as is in the interest of making it available for others to work on if interested.

        Enjoy

        Show
        Mark Ford added a comment - Attached is my camel-sns contribution that has been repackaged by Tracy Snell. This code should be merged with the camel-aws contribution in order to leverage code between the SQS and SNS components. Note that some of the integration tests with SNS are slow due to the delay involved in propagating the permissions for the target queue. At one point the advertised propagation time was 60 seconds but in practice it was over 90 seconds. All of the integration tests that rely on this propagation wait 120 seconds to be sure. A better approach would be to setup a single queue for the integration tests and reuse it for all of the tests. This is left for a future submission (i.e. from Tracy). Additionally, some of the suggestions for the SQS component submission apply here as well. For example, the setting of the Amazon API object, public key, etc would be nice improvements. There are a few FIXME's left undone but they don't affect the functionality. One is a minor tweak and two are missing tests. I'm submitting as is in the interest of making it available for others to work on if interested. Enjoy
        Hide
        Tracy Snell added a comment -

        Christian I'll merge up the two components and submit a new patch.

        Thanks Mark!

        Show
        Tracy Snell added a comment - Christian I'll merge up the two components and submit a new patch. Thanks Mark!
        Hide
        Christian Müller added a comment -

        Ok, sounds good. Looking forward for it...

        Show
        Christian Müller added a comment - Ok, sounds good. Looking forward for it...
        Hide
        Claus Ibsen added a comment -

        Please prefix the component scheme name with aws- so it would be aws-sns.
        This is done by naming the file aws-sns in the META-INF/services/ ... /components directory.

        Show
        Claus Ibsen added a comment - Please prefix the component scheme name with aws- so it would be aws-sns . This is done by naming the file aws-sns in the META-INF/services/ ... /components directory.
        Hide
        Tracy Snell added a comment -

        Made a number of changes to the code. Added several new classes. Improved and added some tests.

        Code coverage is high with integration testing on. More tests I'll add later when I look for areas of reuse between the 2 components. Should build clean and pass sourcecheck.

        Show
        Tracy Snell added a comment - Made a number of changes to the code. Added several new classes. Improved and added some tests. Code coverage is high with integration testing on. More tests I'll add later when I look for areas of reuse between the 2 components. Should build clean and pass sourcecheck.
        Hide
        Claus Ibsen added a comment -

        The keys for the headers can we avoid the colon?

         String SNS_SUBJECT = "SNS:Subject";
        
        Show
        Claus Ibsen added a comment - The keys for the headers can we avoid the colon? String SNS_SUBJECT = "SNS:Subject" ;
        Hide
        Claus Ibsen added a comment -

        And it would be good to avoid system out but using logger, even in unit tests.

        Show
        Claus Ibsen added a comment - And it would be good to avoid system out but using logger, even in unit tests.
        Hide
        Claus Ibsen added a comment -

        @author tags should also be removed.

        Show
        Claus Ibsen added a comment - @author tags should also be removed.
        Hide
        Claus Ibsen added a comment -

        Please adjust logging to a sensible level DEBUG is to verbost. Adjust some of the DEBUG to TRACE level.

        For example debug logging "Starting" doesnt provide much value. Instead log maybe at INFO level that the consumer is starting and listening on XXX endpoint / queue or whatever it listen. Likewise when it stops.

        Camel core itself has logging when it starts/stops consumers/producer etc so we got plenty of generic logging for that.

        Loading resources / classes from classpath should always use the API from CamelContext. There is a ClassResolver. It ensures it can be loaded in OSGI, JBoss and whatever platform

         String s = IOUtils.toString(SnsConsumer.class.getResourceAsStream("/default-sqs-policy-template.json"));
        
        Show
        Claus Ibsen added a comment - Please adjust logging to a sensible level DEBUG is to verbost. Adjust some of the DEBUG to TRACE level. For example debug logging "Starting" doesnt provide much value. Instead log maybe at INFO level that the consumer is starting and listening on XXX endpoint / queue or whatever it listen. Likewise when it stops. Camel core itself has logging when it starts/stops consumers/producer etc so we got plenty of generic logging for that. Loading resources / classes from classpath should always use the API from CamelContext. There is a ClassResolver. It ensures it can be loaded in OSGI, JBoss and whatever platform String s = IOUtils.toString(SnsConsumer.class.getResourceAsStream( "/ default -sqs-policy-template.json" ));
        Hide
        Tracy Snell added a comment -
        • Almost removed the System.out calls from the tests then I wasn't sure if they were allowed or not. Pulling them.
        • Colon for Headers can be removed. I went with Mark's convention there thinking it'd be OK.
        • I saw a few other @author tags in the tree so left it in. Didn't want to be pulling the original author's name if it was allowed.
        • I'll adjust logging
        • Fixing the resource loading.

        If this is not going out in 2.6 I think I'll merge back into Mark's google code repository. He has some test enhancements he'd like to add. Then we can resubmit the patch.

        Show
        Tracy Snell added a comment - Almost removed the System.out calls from the tests then I wasn't sure if they were allowed or not. Pulling them. Colon for Headers can be removed. I went with Mark's convention there thinking it'd be OK. I saw a few other @author tags in the tree so left it in. Didn't want to be pulling the original author's name if it was allowed. I'll adjust logging Fixing the resource loading. If this is not going out in 2.6 I think I'll merge back into Mark's google code repository. He has some test enhancements he'd like to add. Then we can resubmit the patch.
        Hide
        Christian Müller added a comment -

        We will add this new component into Camel 2.7. Hadrian will start building the release in the next couple of days...
        I will have a look on it next week.

        Show
        Christian Müller added a comment - We will add this new component into Camel 2.7. Hadrian will start building the release in the next couple of days... I will have a look on it next week.
        Hide
        Tracy Snell added a comment -

        I've made the changes Claus recommended. I'm going to push the code back into google code so Mark can review and add the improvements he was thinking. Then we'll get a new patch here early in the week. That work?

        Show
        Tracy Snell added a comment - I've made the changes Claus recommended. I'm going to push the code back into google code so Mark can review and add the improvements he was thinking. Then we'll get a new patch here early in the week. That work?
        Hide
        Christian Müller added a comment -

        Perfect.

        Show
        Christian Müller added a comment - Perfect.
        Hide
        Tracy Snell added a comment -

        Should be close now. Give it a look and let me know if I missed anything.

        Show
        Tracy Snell added a comment - Should be close now. Give it a look and let me know if I missed anything.
        Hide
        Tracy Snell added a comment -

        Amazon has changed their public key. I need to add a patch to fix that issue.

        Show
        Tracy Snell added a comment - Amazon has changed their public key. I need to add a patch to fix that issue.
        Hide
        Claus Ibsen added a comment -

        Tracy any update on the patch?

        Show
        Claus Ibsen added a comment - Tracy any update on the patch?
        Hide
        Tracy Snell added a comment -

        Finally not traveling this week. Should be able to make the change and get things updated.

        Show
        Tracy Snell added a comment - Finally not traveling this week. Should be able to make the change and get things updated.
        Hide
        Christian Müller added a comment -

        I'm thinking about the SnsConsumer. In this patch, the SnsConsumer is in fact a SqsConsumer. It polls the messages from a Amazon SQS queue which is subscribed to the SNS topic. The additional functionality in the SnsConsumer is to create a subscribtion from a SQS queue to the SNS topic when we start the SnsConsumer and unsubscribe if we stop the SnsConsumer (and the deleteTopicOnStop option is set to true - I think we can find a better name like unsubscribeOnStop or so). I'm wondering if this is really a functionality we need here. I think most people (or maybe all) will configure the subscription once with the Amazon Management Console and not make the subscription on the fly with Camel.
        I would like to remove the SnsConsumer (or not add) and keep the code as simple as possible because I do not see an useful use case for it.
        What do you think? Any doubts?

        Christian

        Show
        Christian Müller added a comment - I'm thinking about the SnsConsumer. In this patch, the SnsConsumer is in fact a SqsConsumer. It polls the messages from a Amazon SQS queue which is subscribed to the SNS topic. The additional functionality in the SnsConsumer is to create a subscribtion from a SQS queue to the SNS topic when we start the SnsConsumer and unsubscribe if we stop the SnsConsumer (and the deleteTopicOnStop option is set to true - I think we can find a better name like unsubscribeOnStop or so). I'm wondering if this is really a functionality we need here. I think most people (or maybe all) will configure the subscription once with the Amazon Management Console and not make the subscription on the fly with Camel. I would like to remove the SnsConsumer (or not add) and keep the code as simple as possible because I do not see an useful use case for it. What do you think? Any doubts? Christian
        Hide
        Mark Ford added a comment -

        I think there's value in having a way to stand up a subscription to a topic from camel. Imagine if the AMQP or JMS components didn't offer this ability. I know it's slightly different because of how those protocols are implemented but I think this is part of the value of camel's endpoint abstraction. The value add from the use of the component is that it encapsulates the additional administrative details. Changing from Amazon SNS to AMQP should be a simple matter of changing the endpoint URI's in your config and then changing the dependencies. Requiring an additional administrative step to go the Amazon admin console doesn't sound right.

        I don't know how many people would use it this way versus administering it outside of camel but it's nice to have the option.

        Show
        Mark Ford added a comment - I think there's value in having a way to stand up a subscription to a topic from camel. Imagine if the AMQP or JMS components didn't offer this ability. I know it's slightly different because of how those protocols are implemented but I think this is part of the value of camel's endpoint abstraction. The value add from the use of the component is that it encapsulates the additional administrative details. Changing from Amazon SNS to AMQP should be a simple matter of changing the endpoint URI's in your config and then changing the dependencies. Requiring an additional administrative step to go the Amazon admin console doesn't sound right. I don't know how many people would use it this way versus administering it outside of camel but it's nice to have the option.
        Hide
        Tracy Snell added a comment -

        I was mistaken on the public key. It did change but the SNS code grabs a new copy each run so should be good to go there.

        Show
        Tracy Snell added a comment - I was mistaken on the public key. It did change but the SNS code grabs a new copy each run so should be good to go there.
        Hide
        Mark Ford added a comment -

        Doesn't the SNS code cache the AWS key in a static as opposed to "each run"? I think a better approach would be to periodically refresh the key.

        Show
        Mark Ford added a comment - Doesn't the SNS code cache the AWS key in a static as opposed to "each run"? I think a better approach would be to periodically refresh the key.
        Hide
        Christian Müller added a comment -

        I'm still not convinced because of the following reasons:

        • I dislike the code duplication in SnsConsumer.
        • In my opinion, nobody should use the deleteQueueOnStop option in SnsConsumer. It can be result in loosing messages, if the SnsConsumer is down (e.g. for maintenance reasons) and new messages arrives in the SNS topic.
        • And if you don't use the deleteQueueOnStop option, it's useless to make a new subscription each time the SnsConsumer starts.
        • In the Amazon Management Console it's only a few clicks to configure it and I in my opinion, that's different thing than creation the ActiveMQ queues/topics by hand.

        Any other opinions?

        I would like to solve this issue today or tomorrow...

        Show
        Christian Müller added a comment - I'm still not convinced because of the following reasons: I dislike the code duplication in SnsConsumer. In my opinion, nobody should use the deleteQueueOnStop option in SnsConsumer. It can be result in loosing messages, if the SnsConsumer is down (e.g. for maintenance reasons) and new messages arrives in the SNS topic. And if you don't use the deleteQueueOnStop option, it's useless to make a new subscription each time the SnsConsumer starts. In the Amazon Management Console it's only a few clicks to configure it and I in my opinion, that's different thing than creation the ActiveMQ queues/topics by hand. Any other opinions? I would like to solve this issue today or tomorrow...
        Hide
        Mark Ford added a comment -

        The issue of code duplication is orthogonal to the discussion. This is an implementation detail and something that Tracy will likely correct in a future patch.

        As for the loss of messages, wouldn't you lose messages if the consumer was down in other pub/sub frameworks? I know there are such things as durable subscriptions but there are also ephemeral subscriptions as well. The SNSConsumer as it is now supports both.

        As for the deleteQueueOnStop option, I'll admit that I didn't give too much thought to how these options would be used beyond my initial use case which was a project for a graduate course I was taking. I can't defend the particular settings or defaults but I will attempt to defend the concept of a temporary queue. If you think that temporary queues have value, then this and other options will likely need to be reviewed to see how best the semantics should be modeled within the URI options.

        If you require the creation of the subscription through the admin console or something outside of the realm of camel, then I think you're making it more difficult to support dynamic endpoint addressing. My apps are a mix of static and dynamic routes. In some cases contexts are assembled on the fly and routes established to meet the needs of some relatively short lived processing chain. If I needed to support Amazon AWS (which I don't) and I wanted it to work in a dynamic capacity then I would need to include the AWS API into my codebase. This would break the encapsulation I'm getting from camel with respect to nearly all of the other endpoints the app supports (AMQP, UDP, TCP, JMS).

        Show
        Mark Ford added a comment - The issue of code duplication is orthogonal to the discussion. This is an implementation detail and something that Tracy will likely correct in a future patch. As for the loss of messages, wouldn't you lose messages if the consumer was down in other pub/sub frameworks? I know there are such things as durable subscriptions but there are also ephemeral subscriptions as well. The SNSConsumer as it is now supports both. As for the deleteQueueOnStop option, I'll admit that I didn't give too much thought to how these options would be used beyond my initial use case which was a project for a graduate course I was taking. I can't defend the particular settings or defaults but I will attempt to defend the concept of a temporary queue. If you think that temporary queues have value, then this and other options will likely need to be reviewed to see how best the semantics should be modeled within the URI options. If you require the creation of the subscription through the admin console or something outside of the realm of camel, then I think you're making it more difficult to support dynamic endpoint addressing. My apps are a mix of static and dynamic routes. In some cases contexts are assembled on the fly and routes established to meet the needs of some relatively short lived processing chain. If I needed to support Amazon AWS (which I don't) and I wanted it to work in a dynamic capacity then I would need to include the AWS API into my codebase. This would break the encapsulation I'm getting from camel with respect to nearly all of the other endpoints the app supports (AMQP, UDP, TCP, JMS).
        Hide
        Christian Müller added a comment -

        Ok, I will keep this functionality. Still do some housekeeping...

        Show
        Christian Müller added a comment - Ok, I will keep this functionality. Still do some housekeeping...
        Hide
        Christian Müller added a comment -

        Committed r1084013

        Show
        Christian Müller added a comment - Committed r1084013
        Hide
        Christian Müller added a comment -

        @Mark & Tracy: Thank you again for this contribution and your work.

        I polished the patch and applied them yesterday. I only include the SNS producer code, because of my personal opinion and after a few talks with my colleges, I still think configuring the subscription to an Amazon SNS service is not the job we have to solve with Camel. The configuration is done in a few clicks in the AWS Management Console and it sounds not useful for me to develop a Camel route and provide a JSON template to do it.
        Now, our users have the possibility to send messages to an Amazon SNS service and receive messages from an Amazob SQS service. He only has to configure the subscription with the AWS Management Console, which is typical done once.

        Apologze, if you don't share my opinions.
        Christian

        Show
        Christian Müller added a comment - @Mark & Tracy: Thank you again for this contribution and your work. I polished the patch and applied them yesterday. I only include the SNS producer code, because of my personal opinion and after a few talks with my colleges, I still think configuring the subscription to an Amazon SNS service is not the job we have to solve with Camel. The configuration is done in a few clicks in the AWS Management Console and it sounds not useful for me to develop a Camel route and provide a JSON template to do it. Now, our users have the possibility to send messages to an Amazon SNS service and receive messages from an Amazob SQS service. He only has to configure the subscription with the AWS Management Console , which is typical done once. Apologze, if you don't share my opinions. Christian
        Hide
        Mark Ford added a comment -

        Christian,

        I don't share your opinions and find it disappointing and a bad precedent that you've cited the counsel of others that was made off list. I was hoping that these differences would be resolved within the camel community either on this issue or within the developer list. If your colleagues have new views that weren't already addressed then I would have welcomed them here. I don't know what new points I could offer here that weren't already covered above.

        Show
        Mark Ford added a comment - Christian, I don't share your opinions and find it disappointing and a bad precedent that you've cited the counsel of others that was made off list. I was hoping that these differences would be resolved within the camel community either on this issue or within the developer list. If your colleagues have new views that weren't already addressed then I would have welcomed them here. I don't know what new points I could offer here that weren't already covered above.
        Hide
        Hadrian Zbarcea added a comment -

        Mark, Christian,

        I reopened the issue as an attempt to get some consensus here. I respect the fact that you have different views and I feel we'll be able to clarify them quickly.

        I don't have a strong opinion yet, but I get Mark's point. Christian, I am not sure I get your objection.

        Show
        Hadrian Zbarcea added a comment - Mark, Christian, I reopened the issue as an attempt to get some consensus here. I respect the fact that you have different views and I feel we'll be able to clarify them quickly. I don't have a strong opinion yet, but I get Mark's point. Christian, I am not sure I get your objection.
        Hide
        Christian Müller added a comment -

        Sorry Mark, it was not my intention to disappoint you. English is not my first language, so it's a bit harder for me to express me correct. I will take some more time to make my opinions more clear.

        I still think the SNS component should not have a consumer. A consumer make no sense and is not supported by the Amazon API.

        My first reservation is that the SNS consumer had the ability to poll a SQS queue. We already have this functionality in the SNS consumer and there is the right place place for it. I see no reason to have two different components with the same functionality.

        My second reservation is that the SNS consumer had the ability to make the subscription to a SQS queue. What, if the next user want to make a subscription to http(s) or smtp or ... and not to a SQS queue? What is your position to this feature?

        I think the right place to make a subscription (if we want do this) is the SNS producer - not the consumer. If you also agree about this, we could start to discuss whether and how we could implement this (which protocols ...).

        Christian

        Show
        Christian Müller added a comment - Sorry Mark, it was not my intention to disappoint you. English is not my first language, so it's a bit harder for me to express me correct. I will take some more time to make my opinions more clear. I still think the SNS component should not have a consumer. A consumer make no sense and is not supported by the Amazon API. My first reservation is that the SNS consumer had the ability to poll a SQS queue. We already have this functionality in the SNS consumer and there is the right place place for it. I see no reason to have two different components with the same functionality. My second reservation is that the SNS consumer had the ability to make the subscription to a SQS queue. What, if the next user want to make a subscription to http(s) or smtp or ... and not to a SQS queue? What is your position to this feature? I think the right place to make a subscription (if we want do this) is the SNS producer - not the consumer. If you also agree about this, we could start to discuss whether and how we could implement this (which protocols ...). Christian
        Hide
        Mark Ford added a comment -

        I was not disappointed in your decision but only in the lack of discussion. In each one of the exchanges above we appeared to reach consensus and then suddenly you changed your mind based on conversations offline. I appreciate the chance to resolve this with input from others.

        The overlap in functionality doesn't concern me. There are often different ways to accomplish things in Camel and other frameworks. For example, the timer and quartz components offer similar functionality and yet both exist. I would argue that the SQS and SNS components are complementary as opposed to mere duplication.

        I don't understand your comment about a consumer not making sense or not being supported by the Amazon API. First, it makes sense because without consumers, there is no reason to ever publish a message. Second, it is supported by the Amazon API because the Amazon API allows you to create a subscription. The subscription endpoint receives the notification messages and by any definition is a consumer.

        I intentionally did not support the http, http(s), or smtp protocols because I didn't need them for my use case and in fact don't think I would ever use them in a production environment. Most of my Camel deployments live behind a corporate firewall and as such it is difficult to provide an externally available endpoint. I don't have any experience with the SNS platform in production so I don't know how popular the different subscription types are. As an enterprise user, the SQS subscription type that allowed me to poll for messages seemed good enough. If there was a call from other users to support other subscription types, then I'm sure it could be implemented if necessary although this would greatly expand the dependencies and scope of the component. That said, I've seen done this before for a proprietary WS-Notification/WS-Eventing Camel component that leveraged Jetty for the subscription endpoint.

        I don't understand your comment about putting the subscription control into the SNS Producer. It seems weird that you'd create a subscription or check for its creation each time the producer executed.

        The simplest use case I can think of that shows the merit of having an SNS consumer is wanting to have a Camel route where the subscription to the topic is temporary. When the route starts, data flows from the topic into the route as messages become available. When the route stops, the flow of data stops. I don't see how you could support this use case.

        Of course, there are other use cases that may or may not involve using the Amazon Console to create subscriptions out of band. As I see it, the SNS consumer as it exists in the original submission supports all styles.

        • sns consumer with temporary queue
        • sns consumer with permanent queue (by name or ARN)
        • mix of sqs consumer and AWS console

        You're proposing to only support the last in the above list. I think the others have value.

        Finally, I think it makes sense from an end user perspective to try and abstract the subscription details away from a user if possible. I don't really care how I receive the messages from the topic, just as long as I get them and that they're verified as coming from SNS. The SNS consumer provides this functionality by bundling the creation of the subscription and the temporary queue in a single endpoint URI. If you separate them, then you're requiring additional configuration from the end user in terms of the subscription and message processing. The user will then need to configure their routes in advance with the additional SQS endpoint and then possibly provide a custom processor on the route to verify the headers for each message. Again, this code is trivial, but why make them go to the trouble when it could all be done for them?

        Show
        Mark Ford added a comment - I was not disappointed in your decision but only in the lack of discussion. In each one of the exchanges above we appeared to reach consensus and then suddenly you changed your mind based on conversations offline. I appreciate the chance to resolve this with input from others. The overlap in functionality doesn't concern me. There are often different ways to accomplish things in Camel and other frameworks. For example, the timer and quartz components offer similar functionality and yet both exist. I would argue that the SQS and SNS components are complementary as opposed to mere duplication. I don't understand your comment about a consumer not making sense or not being supported by the Amazon API. First, it makes sense because without consumers, there is no reason to ever publish a message. Second, it is supported by the Amazon API because the Amazon API allows you to create a subscription. The subscription endpoint receives the notification messages and by any definition is a consumer. I intentionally did not support the http, http(s), or smtp protocols because I didn't need them for my use case and in fact don't think I would ever use them in a production environment. Most of my Camel deployments live behind a corporate firewall and as such it is difficult to provide an externally available endpoint. I don't have any experience with the SNS platform in production so I don't know how popular the different subscription types are. As an enterprise user, the SQS subscription type that allowed me to poll for messages seemed good enough. If there was a call from other users to support other subscription types, then I'm sure it could be implemented if necessary although this would greatly expand the dependencies and scope of the component. That said, I've seen done this before for a proprietary WS-Notification/WS-Eventing Camel component that leveraged Jetty for the subscription endpoint. I don't understand your comment about putting the subscription control into the SNS Producer. It seems weird that you'd create a subscription or check for its creation each time the producer executed. The simplest use case I can think of that shows the merit of having an SNS consumer is wanting to have a Camel route where the subscription to the topic is temporary. When the route starts, data flows from the topic into the route as messages become available. When the route stops, the flow of data stops. I don't see how you could support this use case. Of course, there are other use cases that may or may not involve using the Amazon Console to create subscriptions out of band. As I see it, the SNS consumer as it exists in the original submission supports all styles. sns consumer with temporary queue sns consumer with permanent queue (by name or ARN) mix of sqs consumer and AWS console You're proposing to only support the last in the above list. I think the others have value. Finally, I think it makes sense from an end user perspective to try and abstract the subscription details away from a user if possible. I don't really care how I receive the messages from the topic, just as long as I get them and that they're verified as coming from SNS. The SNS consumer provides this functionality by bundling the creation of the subscription and the temporary queue in a single endpoint URI. If you separate them, then you're requiring additional configuration from the end user in terms of the subscription and message processing. The user will then need to configure their routes in advance with the additional SQS endpoint and then possibly provide a custom processor on the route to verify the headers for each message. Again, this code is trivial, but why make them go to the trouble when it could all be done for them?
        Hide
        Christian Müller added a comment -

        It looks like I was too impatient...

        To clarify the naming:

        • SQS producer: A class which extends the DefaultProducer to implement the event driven consumer enterprise integration pattern. Used to consume message exchanges and send it to the Amazon SQS service (SendMessageRequest).
        • SQS consumer: A class which extends the ScheduledPollConsumer to poll messages from an Amazon SQS service (ReceiveMessageRequest) and create message exchanges from this.
        • SNS producer: A class which extends the DefaultProducer to implement the event driven consumer enterprise integration pattern. Used to consume message exchanges and send it to the Amazon SNS service (PublishRequest).
        • SNS consumer (not exists yet): A class which extends the DefaultConsumer (or ScheduledPollConsumer) to receive (or poll) messages from an Amazon SQS service and create message exchanges from this.

        Because the AmazonSNSClient doesn't offer a readXXX(), receiveXXX(), ... method to receive messages, I think a SNS consumer make no sense. Creating a subscription to an Amazon SNS service/topic has nothing to do with with consuming messages from this endpoint and so it's not a SNS consumer.

        The subscription can be made to an Amazon SQS queue, to an HTTP(S) endpoint or to an e-Mail. For all this protocols/endpoints, Camel already offer consumers.

        The only thing Camel at present do not support is the subscription to an Amazon SNS service/topic. If we really want/need this, I think the SNS producer should support this. He can do this in the doStart() method and not each time a message exchange receive (creating the SNS topic, creating the SQS queue, making the subscription). And in your second route you use the SQS consumer to consume the messages.

        As you say, the timer and quartz component offer similar functionality. But if the SNS consumer consumes messages from an Amazob SQS queue, the SNS consumer and SQS consumer offer the same functionality. This makes no sense.

        I don't understand your last paragraph. I think you cannot "hide" or abstract the subscription. The user which receives the messages, send to an Amazon SNS service/topic has to know, where the messages are forwarded to.

        And I think the subscription is not so easy as you think. In my opinion, the value of this Amazon SNS service is to broadcast one message to multiple recipients (and not to work around a firewall issue). How would you handle/implement multiple subscriptions? Some subscriptions like e-Mail requires an acknowledgement (a response mail of the subscription mail) before a message is forwarded. If this component cannot handle these, I think most (if not all) of the subscriptions must be made by hand.

        Christian

        Show
        Christian Müller added a comment - It looks like I was too impatient... To clarify the naming: SQS producer: A class which extends the DefaultProducer to implement the event driven consumer enterprise integration pattern. Used to consume message exchanges and send it to the Amazon SQS service (SendMessageRequest). SQS consumer: A class which extends the ScheduledPollConsumer to poll messages from an Amazon SQS service (ReceiveMessageRequest) and create message exchanges from this. SNS producer: A class which extends the DefaultProducer to implement the event driven consumer enterprise integration pattern. Used to consume message exchanges and send it to the Amazon SNS service (PublishRequest). SNS consumer (not exists yet): A class which extends the DefaultConsumer (or ScheduledPollConsumer) to receive (or poll) messages from an Amazon SQS service and create message exchanges from this. Because the AmazonSNSClient doesn't offer a readXXX(), receiveXXX(), ... method to receive messages, I think a SNS consumer make no sense. Creating a subscription to an Amazon SNS service/topic has nothing to do with with consuming messages from this endpoint and so it's not a SNS consumer. The subscription can be made to an Amazon SQS queue, to an HTTP(S) endpoint or to an e-Mail. For all this protocols/endpoints, Camel already offer consumers. The only thing Camel at present do not support is the subscription to an Amazon SNS service/topic. If we really want/need this, I think the SNS producer should support this. He can do this in the doStart() method and not each time a message exchange receive (creating the SNS topic, creating the SQS queue, making the subscription). And in your second route you use the SQS consumer to consume the messages. As you say, the timer and quartz component offer similar functionality. But if the SNS consumer consumes messages from an Amazob SQS queue, the SNS consumer and SQS consumer offer the same functionality. This makes no sense. I don't understand your last paragraph. I think you cannot "hide" or abstract the subscription. The user which receives the messages, send to an Amazon SNS service/topic has to know, where the messages are forwarded to. And I think the subscription is not so easy as you think. In my opinion, the value of this Amazon SNS service is to broadcast one message to multiple recipients (and not to work around a firewall issue). How would you handle/implement multiple subscriptions? Some subscriptions like e-Mail requires an acknowledgement (a response mail of the subscription mail) before a message is forwarded. If this component cannot handle these, I think most (if not all) of the subscriptions must be made by hand. Christian
        Hide
        Mark Ford added a comment -

        I don't understand why the producer would create the subscription. Why would you tightly couple the production of messages with the consumption of messages?

        The duplication of code is irrelevant. Tracy can refactor this so the SNS consumer is nothing more than an instantiation of the SQS consumer.

        Multiple subscribers already work with the SNS component as it is. Each route would create the from with the same topic name or ARN and then use their own SQS endpoint (by name or ARN) to receive the messages.

        I think I've already encapsulated the details of the subscription pretty well in the component. When I am creating a route to read messages from an endpoint, I am only focused on the messages I'm receiving and not how I am receiving them. These details are encapsulated in the component with the creation of a temporary queue. With your model, you must configure the subscription externally.

        Show
        Mark Ford added a comment - I don't understand why the producer would create the subscription. Why would you tightly couple the production of messages with the consumption of messages? The duplication of code is irrelevant. Tracy can refactor this so the SNS consumer is nothing more than an instantiation of the SQS consumer. Multiple subscribers already work with the SNS component as it is. Each route would create the from with the same topic name or ARN and then use their own SQS endpoint (by name or ARN) to receive the messages. I think I've already encapsulated the details of the subscription pretty well in the component. When I am creating a route to read messages from an endpoint, I am only focused on the messages I'm receiving and not how I am receiving them. These details are encapsulated in the component with the creation of a temporary queue. With your model, you must configure the subscription externally.
        Hide
        Christian Müller added a comment -

        I don't agree, the SNS consumer is not only a "SQS consumer". It must be also a HTTP/Jetty consumer and a Mail consumer to support all subscriptions.

        However, I think we have two different position and both are probably valid. But I think we make no progress here.

        @Hadrian: You reopened this issue (what was right). What are your thoughts?
        @Claus+Willem: You also shared your thoughts with me/us. What you are thinking now?
        @all others: What you have in mind?

        Christian

        Show
        Christian Müller added a comment - I don't agree, the SNS consumer is not only a "SQS consumer". It must be also a HTTP/Jetty consumer and a Mail consumer to support all subscriptions. However, I think we have two different position and both are probably valid. But I think we make no progress here. @Hadrian: You reopened this issue (what was right). What are your thoughts? @Claus+Willem: You also shared your thoughts with me/us. What you are thinking now? @all others: What you have in mind? Christian
        Hide
        Mark Ford added a comment -

        I think we've made a good deal of progress. The only point you've raised for which I don't have an answer is the lack of support for http(s) and email. I have no plans on implementing these subscription models. Perhaps Tracy or someone else does.

        If the consensus is that components must offer all or no functionality then I can see where you would view the submission as a partial implementation and reject it. I can accept that and leave it to the community to implement all of the subscription models if desired.

        Show
        Mark Ford added a comment - I think we've made a good deal of progress. The only point you've raised for which I don't have an answer is the lack of support for http(s) and email. I have no plans on implementing these subscription models. Perhaps Tracy or someone else does. If the consensus is that components must offer all or no functionality then I can see where you would view the submission as a partial implementation and reject it. I can accept that and leave it to the community to implement all of the subscription models if desired.
        Hide
        Tracy Snell added a comment -

        My plan has always been to add support for email and http/https to the consumer. Mark wrote the SNS code independently (and before) I wrote the SQS code. First pass was just to get it into the component then add support for the other two protocols so in the end the SNS consumer would delegate all three and handle the message format.

        Show
        Tracy Snell added a comment - My plan has always been to add support for email and http/https to the consumer. Mark wrote the SNS code independently (and before) I wrote the SQS code. First pass was just to get it into the component then add support for the other two protocols so in the end the SNS consumer would delegate all three and handle the message format.
        Hide
        Hadrian Zbarcea added a comment -

        I also think we made good progress and we'll sort this one out.

        I think Mark is more focused on the problem, and I agree with his assessment. Christian seems to have objections related mostly to the solution, and to me it only means that we need to find a better one.

        I also believe that while the SNS consumer it not a listener on a particular protocol, which is what Christian pointed out, it does however make sense to talk about an SNS consumer. The SNS consumer sits at the receiving end of the event driven consumer eip and handles messages coming to an SNS endpoint. The fact that it has to delegate to protocol specific consumers for part of the job is an implementation detail. The hardest part will be imho to get the definition of the sns endpoint URI right.

        Mark, it seems, provided a solution to his immediate problem, which sparked the discussion, In his last comment though he seems to agree that the solution could/should be more general, however the non SQS stuff is not on his radar and he'll leave it to the community to implement that. Looks like Tracy will come up with something what will make everybody happy. Tracy, does it brew coffee too?

        Show
        Hadrian Zbarcea added a comment - I also think we made good progress and we'll sort this one out. I think Mark is more focused on the problem, and I agree with his assessment. Christian seems to have objections related mostly to the solution, and to me it only means that we need to find a better one. I also believe that while the SNS consumer it not a listener on a particular protocol, which is what Christian pointed out, it does however make sense to talk about an SNS consumer. The SNS consumer sits at the receiving end of the event driven consumer eip and handles messages coming to an SNS endpoint. The fact that it has to delegate to protocol specific consumers for part of the job is an implementation detail. The hardest part will be imho to get the definition of the sns endpoint URI right. Mark, it seems, provided a solution to his immediate problem, which sparked the discussion, In his last comment though he seems to agree that the solution could/should be more general, however the non SQS stuff is not on his radar and he'll leave it to the community to implement that. Looks like Tracy will come up with something what will make everybody happy. Tracy, does it brew coffee too?
        Hide
        Christian Müller added a comment -

        There is no all or nothing rule I know. I only try to think one step further...

        The next issue could be, if the owner of the SNS topic is another as the SQS queue. Than you have to have both credentials to set up the subscription...

        Show
        Christian Müller added a comment - There is no all or nothing rule I know. I only try to think one step further... The next issue could be, if the owner of the SNS topic is another as the SQS queue. Than you have to have both credentials to set up the subscription...
        Hide
        Mark Ford added a comment -

        I don't think you need two sets of credentials. Assume that User-1 creates the topic. If User-2 wishes to subscribe to the topic, then they need to create the route with the appropriate URI that includes the ARN of the topic. The newly created SQS queue will be owned by User-2 and not require the credentials for User-1. The permissions on the newly created queue are configured to allow the messages from the topic ARN to arrive.

        Note: I have not tested with multiple AWS accounts.

        One assumption is that the topic setup by User-1 is configured to allow User-2 to subscribe. This may be done a number of ways through the admin console. I could see where it would be nice for the component to configure the topic permissions during the creation of the topic. Perhaps this is something the producer could do.

        In either case, I don't see needing multiple credentials here.

        Show
        Mark Ford added a comment - I don't think you need two sets of credentials. Assume that User-1 creates the topic. If User-2 wishes to subscribe to the topic, then they need to create the route with the appropriate URI that includes the ARN of the topic. The newly created SQS queue will be owned by User-2 and not require the credentials for User-1. The permissions on the newly created queue are configured to allow the messages from the topic ARN to arrive. Note: I have not tested with multiple AWS accounts. One assumption is that the topic setup by User-1 is configured to allow User-2 to subscribe. This may be done a number of ways through the admin console. I could see where it would be nice for the component to configure the topic permissions during the creation of the topic. Perhaps this is something the producer could do. In either case, I don't see needing multiple credentials here.
        Hide
        Christian Müller added a comment -

        Tracy, because you will work further on this issue, you should be the ticket holder. ;o)
        I will watch this issue and review the further contributions...

        Show
        Christian Müller added a comment - Tracy, because you will work further on this issue, you should be the ticket holder. ;o) I will watch this issue and review the further contributions...
        Hide
        Mark Ford added a comment -

        I think the addition of the consumer to the API should split out to its own issue since 2.8.0 will include the producer. I recall seeing some traffic on the dev/user list with a use case request that would be addressed by the addition of the consumer so I think there's still value in having an sns consumer.

        Show
        Mark Ford added a comment - I think the addition of the consumer to the API should split out to its own issue since 2.8.0 will include the producer. I recall seeing some traffic on the dev/user list with a use case request that would be addressed by the addition of the consumer so I think there's still value in having an sns consumer.
        Hide
        Kai Wähner added a comment -

        Hey guys,

        one great thing about Camel is to do very easy integrations within routes. You can (but you do not have to) specify almost all configurations in the endpoints. This is awesome if you need to change an endpoint to another technology - also for playing around, evaluating and testing technologies.

        I have played around with the camel-aws component this week. In my opinion, there should be a consumer which is able to subscribe to topics, otherwise Camel is missing a key part of the AWS SNS service!
        I have written a blog about the camel-aws component and mentioned this discussion (http://www.kai-waehner.de/blog/2011/08/30/cloud-integration-with-apache-camel-and-amazon-web-services-aws-s3-sqs-and-sns). Maybe more people will contribute their opinion.

        Best regards,
        Kai

        Show
        Kai Wähner added a comment - Hey guys, one great thing about Camel is to do very easy integrations within routes. You can (but you do not have to) specify almost all configurations in the endpoints. This is awesome if you need to change an endpoint to another technology - also for playing around, evaluating and testing technologies. I have played around with the camel-aws component this week. In my opinion, there should be a consumer which is able to subscribe to topics, otherwise Camel is missing a key part of the AWS SNS service! I have written a blog about the camel-aws component and mentioned this discussion ( http://www.kai-waehner.de/blog/2011/08/30/cloud-integration-with-apache-camel-and-amazon-web-services-aws-s3-sqs-and-sns ). Maybe more people will contribute their opinion. Best regards, Kai
        Hide
        Christian Müller added a comment -

        Kai, I added a link from our Artices site to your blog.

        Show
        Christian Müller added a comment - Kai, I added a link from our Artices site to your blog.
        Hide
        Christian Müller added a comment -

        Kai, I looking forward for the community feedback you are asking for.
        In the meantime, you have to configure the subscription once via the AWS web console. Than you can use a Camel consumer (SQS, Mail, HTTP or HTTP4) to consume the messages.

        And as you probably know, we love contributions...

        Best,
        Christian

        Show
        Christian Müller added a comment - Kai, I looking forward for the community feedback you are asking for. In the meantime, you have to configure the subscription once via the AWS web console. Than you can use a Camel consumer (SQS, Mail, HTTP or HTTP4) to consume the messages. And as you probably know, we love contributions... Best, Christian

          People

          • Assignee:
            Tracy Snell
            Reporter:
            Mark Ford
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development