Camel
  1. Camel
  2. CAMEL-4256

Adding a EndpointConfiguration interface

    Details

    • Type: New Feature New Feature
    • Status: In Progress
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 2.8.0
    • Fix Version/s: None
    • Component/s: camel-core
    • Labels:
      None
    • Patch Info:
      Patch Available

      Description

      One of the key missing pieces from the API is the explicit concept of EndpointConfiguration. We use URIs for that, ant that's great, but we don't have it in the API. Some components do have an informal version though.

      I am proposing adding an EndpointConfiguration interface:

      public interface EndpointConfiguration {
        void fromUri(String uri);
        String toUri();
      }
      

      and maybe other methods, we could also use URI instead of string for pre-parsing. Same as with other concepts the default implementation would be in impl and components would extend that and add fields for configuration parameters.
      This would solve problems related to URI uniqueness to a good extent as toUri() should always place parameters in the same order. The Endpoint interface would change though.

      The main advantage would be that we can annotate parameters and use javax.validation to specify if a field is @ProducerOnly, @ConsumerOnly for example, which may exclude them from toUri() (yes, there are some impacts, the id uri would be different than the config uric). We could annotate them with @Secret to indicate that at least the value should not appear in clear in the uri, etc. We could also add an @Default("value"), allowing us to exclude from the uri fields set on the default value (even if the filed was explicitly set) and so on.

      This would also make static validation possible unit testing configuration would be vastly simplified and we could improve coverage. We can do it in an incremental way without a big impact on existing components (especially outside camel) via changes in DefaultEndpoint. I am working on a prototype, but feedback is highly appreciated.

      1. CAMEL-4256-test.diff
        4 kB
        Łukasz Dywicki
      2. CAMEL-4256-rmi.diff
        25 kB
        Łukasz Dywicki
      3. CAMEL-4256-core.diff
        4 kB
        Łukasz Dywicki
      4. CAMEL-4256-2.patch
        2 kB
        Keith Babo
      5. camel-4256.patch
        30 kB
        Hadrian Zbarcea

        Activity

        Hide
        Hadrian Zbarcea added a comment -

        Lukasz, I think there's something missing in your new core patch. Can you please reattach? Thanks.

        Show
        Hadrian Zbarcea added a comment - Lukasz, I think there's something missing in your new core patch. Can you please reattach? Thanks.
        Hide
        Hadrian Zbarcea added a comment -

        The core and test patches look good. Looking at rmi now.

        Being able to validate configurations should be a camel feature usable by tools, not only in camel-test. It cannot be in the core because of the extra dependencies, but we'll need at least an interface and a default impl (that may not do much) and a more serious jsr303 impl in a separate jar.

        Thanks for the contributions. We expect much more from you!

        Show
        Hadrian Zbarcea added a comment - The core and test patches look good. Looking at rmi now. Being able to validate configurations should be a camel feature usable by tools, not only in camel-test. It cannot be in the core because of the extra dependencies, but we'll need at least an interface and a default impl (that may not do much) and a more serious jsr303 impl in a separate jar. Thanks for the contributions. We expect much more from you!
        Hide
        Łukasz Dywicki added a comment -

        Re-attach core patch with grant for inclusion. Small changes in EndpointConfiguration interface. Usage of IntrospectionSupport in ConfigurationHelper. New constructor for DefaultEndpoint.

        Show
        Łukasz Dywicki added a comment - Re-attach core patch with grant for inclusion. Small changes in EndpointConfiguration interface. Usage of IntrospectionSupport in ConfigurationHelper. New constructor for DefaultEndpoint.
        Hide
        Łukasz Dywicki added a comment -

        RMI component refactored to use EndpointConfiguration. Attached patch contains also unit test changes.

        Show
        Łukasz Dywicki added a comment - RMI component refactored to use EndpointConfiguration. Attached patch contains also unit test changes.
        Hide
        Łukasz Dywicki added a comment -

        Base class for tests configuration tests. It uses JSR 303 to validate objects. Used implementation is Apache BVAL.

        Show
        Łukasz Dywicki added a comment - Base class for tests configuration tests. It uses JSR 303 to validate objects. Used implementation is Apache BVAL.
        Hide
        Łukasz Dywicki added a comment -

        Small changes in EndpointConfiguration interface. Usage of IntrospectionSupport in ConfigurationHelper. New constructor for DefaultEndpoint.

        Show
        Łukasz Dywicki added a comment - Small changes in EndpointConfiguration interface. Usage of IntrospectionSupport in ConfigurationHelper. New constructor for DefaultEndpoint.
        Hide
        Hadrian Zbarcea added a comment - - edited

        Of course there is. There is one more patch I have for this issue before we can close it. The good part is that everything is transparent and backwards compatible.

        I will add the documentation towards the end of the week, I am @apachecon this week with quite a few things to do.

        Show
        Hadrian Zbarcea added a comment - - edited Of course there is. There is one more patch I have for this issue before we can close it. The good part is that everything is transparent and backwards compatible. I will add the documentation towards the end of the week, I am @apachecon this week with quite a few things to do.
        Hide
        Claus Ibsen added a comment -

        Is there not any documentation that is needed?

        Show
        Claus Ibsen added a comment - Is there not any documentation that is needed?
        Hide
        Keith Babo added a comment -

        Submitting a patch which provides an implementation of createConfiguration() in DefaultComponent. Now DefaultEndpoint can ask the component to produce the configuration object vs. creating it directly in DefaultEndpoint.

        Show
        Keith Babo added a comment - Submitting a patch which provides an implementation of createConfiguration() in DefaultComponent. Now DefaultEndpoint can ask the component to produce the configuration object vs. creating it directly in DefaultEndpoint.
        Hide
        Claus Ibsen added a comment -

        I can see from one point a value in a light weight POJO configuration class.

        However as James also says, ppl can configure endpoints in any way they like

        • uris
        • IoC
        • Java code with new XXX and then invoke the appropriate setters
        • etc.

        And I guess there are situations where you cannot fully configure using uris solely. However I guess 95%+ currently can.
        And most end users use uri configurations as of today.

        If every Endpoint have a default no-arg ctr, then you could also consider the endpoint as a POJO, which have getter/setter for all the options it support. And thus also from a tooling point of view, it would be just as easy to configure, as a xxxConfiguration class would be. Creating an Endpoint using a default no-arg ctr, will of course not cause the Endpoint to be enlisted in registry and whatnot. And since its not started using start() it does not have side-effects.

        Some of the Camel components have a configuration POJO such as the JmsConfiguration, MailConfiguration etc. The latter has a clone() which is needed when people can overlay default configurations by overriding options in endpoint uris. So each endpoint need to clone the default xxxConfiguration. And then set the options from the endpoint uri. This allows people to configure default configuration on the xxxComponent, and then still be able to adjust by overriding in endpoint uris.

        Show
        Claus Ibsen added a comment - I can see from one point a value in a light weight POJO configuration class. However as James also says, ppl can configure endpoints in any way they like uris IoC Java code with new XXX and then invoke the appropriate setters etc. And I guess there are situations where you cannot fully configure using uris solely. However I guess 95%+ currently can. And most end users use uri configurations as of today. If every Endpoint have a default no-arg ctr, then you could also consider the endpoint as a POJO, which have getter/setter for all the options it support. And thus also from a tooling point of view, it would be just as easy to configure, as a xxxConfiguration class would be. Creating an Endpoint using a default no-arg ctr, will of course not cause the Endpoint to be enlisted in registry and whatnot. And since its not started using start() it does not have side-effects. Some of the Camel components have a configuration POJO such as the JmsConfiguration, MailConfiguration etc. The latter has a clone() which is needed when people can overlay default configurations by overriding options in endpoint uris. So each endpoint need to clone the default xxxConfiguration. And then set the options from the endpoint uri. This allows people to configure default configuration on the xxxComponent, and then still be able to adjust by overriding in endpoint uris.
        Hide
        Claus Ibsen added a comment -

        A new method was added to Component

        • EndpointConfiguration createConfiguration(String uri) throws Exception;

        However I guess Camel component developers may extend DefaultComponent instead of implementing the Component interface.
        Likewise a new method was added to Endpoint, however as Endpoint have many methods, then I assume most people would extend DefaultEndpoint, to avoid having to implement all those methods.

        Show
        Claus Ibsen added a comment - A new method was added to Component EndpointConfiguration createConfiguration(String uri) throws Exception; However I guess Camel component developers may extend DefaultComponent instead of implementing the Component interface. Likewise a new method was added to Endpoint, however as Endpoint have many methods, then I assume most people would extend DefaultEndpoint, to avoid having to implement all those methods.
        Hide
        Hadrian Zbarcea added a comment -

        @James, not sure if you're talking about the name of the method toUri or the functionality. Converting back to a uri is being done for ages by normalize. Could we find a better name, probably. There is also another purpose, the uri actually represents both the id and configuration of a communication channel. The issue is that the ends of the channel may be configured differently, because of parameters that are only relevant at one end. The logic of what the uri should look like at either end should be embedded in the EndpointConfiguration.

        Show
        Hadrian Zbarcea added a comment - @James, not sure if you're talking about the name of the method toUri or the functionality. Converting back to a uri is being done for ages by normalize. Could we find a better name, probably. There is also another purpose, the uri actually represents both the id and configuration of a communication channel. The issue is that the ends of the channel may be configured differently, because of parameters that are only relevant at one end. The logic of what the uri should look like at either end should be embedded in the EndpointConfiguration.
        Hide
        Hadrian Zbarcea added a comment -

        @Claus, please point out the problems and I will fix them. I introduce the preProcess method especially to support using the old configuration. The goal is indeed to fully support the old configuration.

        To your other question, why a new interface? That's easy. The relation between an Endpoint and its configuration is not an 'is a' relationship, but a 'has a' relationship. More clearly (if needed) an Endpoint has a configuration. There is more of an is a relationship between a URI and EndpointConfiguration, but not really, it's more about representation than inheritance.

        That allows one (tools for instance) to create and validate an EndpointConfiguration without creating an Endoint that servers another purpose and will end up in the context registry etc. Just basic design, stuff. We ourselves could use it, and I am planning on getting there, to have better coverage in our components of configuration combinations, obviously without the unnecessary creation of all the endpoints and routes that are sooo time consuming.

        If you have more questions, don't hesitate to ask.

        Show
        Hadrian Zbarcea added a comment - @Claus, please point out the problems and I will fix them. I introduce the preProcess method especially to support using the old configuration. The goal is indeed to fully support the old configuration. To your other question, why a new interface? That's easy. The relation between an Endpoint and its configuration is not an 'is a' relationship, but a 'has a' relationship. More clearly (if needed) an Endpoint has a configuration. There is more of an is a relationship between a URI and EndpointConfiguration, but not really, it's more about representation than inheritance. That allows one (tools for instance) to create and validate an EndpointConfiguration without creating an Endoint that servers another purpose and will end up in the context registry etc. Just basic design, stuff. We ourselves could use it, and I am planning on getting there, to have better coverage in our components of configuration combinations, obviously without the unnecessary creation of all the endpoints and routes that are sooo time consuming. If you have more questions, don't hesitate to ask.
        Hide
        Claus Ibsen added a comment -

        @Claus, this is a good place to discuss. Add your comments and suggestions here. This solution is backwards compatible and changes nothing for 3rd party components (already developed or not). It solves a couple of problems and introduces the validation feature. How would you make it more general?

        The code which is committed is not backwards compatible.

        Show
        Claus Ibsen added a comment - @Claus, this is a good place to discuss. Add your comments and suggestions here. This solution is backwards compatible and changes nothing for 3rd party components (already developed or not). It solves a couple of problems and introduces the validation feature. How would you make it more general? The code which is committed is not backwards compatible.
        Hide
        james strachan added a comment -

        Endpoints are already beans you can configure yourself using Spring, Guice, CDI, Java code or whatever and wire them into a CamelContext.
        Dependency injection frameworks can already configure Endpoints today. No need for a new magic interface and yet more work for component/endpoint developers to have to create (which can soon get out of sync with the actual implementation code).

        Tools can use introspection already to figure out how to configure an Endpoint. Just check out how Spring does this or tools based around Spring.

        There could be some added annotations we could use to make it easier to add extra metadata to tools; though where possible I think its better for the community to stick to standards that exist around DI (CDI / Spring / Guice) or around things like the Bean Validation Framework for adding things like min/max validation annotations. Those annotations are already well defined - lets not try reinvent DI / validation wheels.
        http://jcp.org/en/jsr/detail?id=303

        I'd say tools either go the URL option, or go the bean configuration option using (say) spring/guice/CDI. The URI is always going to be a subset of the actual configuration; if you really want to support full endpoint configuration then the only real option is the full DI route; as there are things you can configure though DI tooling you can't through URIs.

        In terms of the EndpointConfiguration interface, fromUri() makes no sense - since that is what a Component does. (Its really its main responsibility). There's no point forcing implementors to do the same thing twice in 2 separate APIs.

        In terms of toUri() for an Endpoint; this is of dubious value IMHO; since its easily possible to create an Endpoint, configure it and then not be able to create a URI which truly reflects the configuration. Its way too easy to loose information; so why even bother trying to turn an Endpoint bean back into a URI? Just use the Endpoint object directly. Taking an Endpoint bean thats been configured somehow and turning it back into a URI just seems to be asking for trouble - when there's no real need to do that - just reuse the Endpoint object as is.

        Show
        james strachan added a comment - Endpoints are already beans you can configure yourself using Spring, Guice, CDI, Java code or whatever and wire them into a CamelContext. Dependency injection frameworks can already configure Endpoints today. No need for a new magic interface and yet more work for component/endpoint developers to have to create (which can soon get out of sync with the actual implementation code). Tools can use introspection already to figure out how to configure an Endpoint. Just check out how Spring does this or tools based around Spring. There could be some added annotations we could use to make it easier to add extra metadata to tools; though where possible I think its better for the community to stick to standards that exist around DI (CDI / Spring / Guice) or around things like the Bean Validation Framework for adding things like min/max validation annotations. Those annotations are already well defined - lets not try reinvent DI / validation wheels. http://jcp.org/en/jsr/detail?id=303 I'd say tools either go the URL option, or go the bean configuration option using (say) spring/guice/CDI. The URI is always going to be a subset of the actual configuration; if you really want to support full endpoint configuration then the only real option is the full DI route; as there are things you can configure though DI tooling you can't through URIs. In terms of the EndpointConfiguration interface, fromUri() makes no sense - since that is what a Component does. (Its really its main responsibility). There's no point forcing implementors to do the same thing twice in 2 separate APIs. In terms of toUri() for an Endpoint; this is of dubious value IMHO; since its easily possible to create an Endpoint, configure it and then not be able to create a URI which truly reflects the configuration. Its way too easy to loose information; so why even bother trying to turn an Endpoint bean back into a URI? Just use the Endpoint object directly. Taking an Endpoint bean thats been configured somehow and turning it back into a URI just seems to be asking for trouble - when there's no real need to do that - just reuse the Endpoint object as is.
        Hide
        Claus Ibsen added a comment -

        Why introduce an interface? You could use those annotations in the Endpoint itself? Why force people to use a xxxConfiguration class?

        Show
        Claus Ibsen added a comment - Why introduce an interface? You could use those annotations in the Endpoint itself? Why force people to use a xxxConfiguration class?
        Hide
        Claus Ibsen added a comment -

        A post on @dev with [DISCUSS] has the tendency to bring in more people, as it stand out on that mailing list, among the hundres of JIRA updates its posted on @dev as well.

        This is after all a topic on camel-core and the Component/Endpoint API which is a very important API in Camel. So I would prefer the great good of the Camel team is behind this.

        Show
        Claus Ibsen added a comment - A post on @dev with [DISCUSS] has the tendency to bring in more people, as it stand out on that mailing list, among the hundres of JIRA updates its posted on @dev as well. This is after all a topic on camel-core and the Component/Endpoint API which is a very important API in Camel. So I would prefer the great good of the Camel team is behind this.
        Hide
        Hadrian Zbarcea added a comment -

        @Claus, this is a good place to discuss. Add your comments and suggestions here. This solution is backwards compatible and changes nothing for 3rd party components (already developed or not). It solves a couple of problems and introduces the validation feature. How would you make it more general?

        Show
        Hadrian Zbarcea added a comment - @Claus, this is a good place to discuss. Add your comments and suggestions here. This solution is backwards compatible and changes nothing for 3rd party components (already developed or not). It solves a couple of problems and introduces the validation feature. How would you make it more general?
        Hide
        Hadrian Zbarcea added a comment -

        @Keith, yes, adding the configuration should be as easy as adding a pojo with annotated fields. A sensible default implementation for instantiation, validation, and serialization to/from URI String is provided by the core runtime (via helpers in ConfigurationHelper).

        Show
        Hadrian Zbarcea added a comment - @Keith, yes, adding the configuration should be as easy as adding a pojo with annotated fields. A sensible default implementation for instantiation, validation, and serialization to/from URI String is provided by the core runtime (via helpers in ConfigurationHelper).
        Hide
        Claus Ibsen added a comment -

        I would suggest to start a [DISCUSS] on dev about this. It seems there is a more broad perspective to this. For example you can configure EIPs, components, endpoints, consumers, producers, data formats, etc. So we may want to take the time to look into a long term and more general solution.

        Also for 3rd party component developers its been so easy so far, as all they have to care about is Component, Endpoint, and Producer/Consumer. So we should ensure it still is dead simple to create custom components. I would really dislike if any API changes causes this to be more complicated.

        Show
        Claus Ibsen added a comment - I would suggest to start a [DISCUSS] on dev about this. It seems there is a more broad perspective to this. For example you can configure EIPs, components, endpoints, consumers, producers, data formats, etc. So we may want to take the time to look into a long term and more general solution. Also for 3rd party component developers its been so easy so far, as all they have to care about is Component, Endpoint, and Producer/Consumer. So we should ensure it still is dead simple to create custom components. I would really dislike if any API changes causes this to be more complicated.
        Hide
        Keith Babo added a comment -

        The approach to start with a default, map-based implementation across components and then updating each component to have a bean-based configuration object seems reasonable to me. The idea of annotating configuration fields to declare whether they are consumer or provider oriented is nice for producing URIs based on the context they are used (using UriFormat) and also documenting the intent of each field. If I understand the approach correctly, it should be possible to use something like ConfigurationHelper to provide a single implementation of turning a config object into a URI string (i.e. each component won't have to implement toURIString() individually) and the rules for how parameters are represented (e.g. query parameters are listed in alphabetical order) would be consistent across all components. Did I get that right?

        Show
        Keith Babo added a comment - The approach to start with a default, map-based implementation across components and then updating each component to have a bean-based configuration object seems reasonable to me. The idea of annotating configuration fields to declare whether they are consumer or provider oriented is nice for producing URIs based on the context they are used (using UriFormat) and also documenting the intent of each field. If I understand the approach correctly, it should be possible to use something like ConfigurationHelper to provide a single implementation of turning a config object into a URI string (i.e. each component won't have to implement toURIString() individually) and the rules for how parameters are represented (e.g. query parameters are listed in alphabetical order) would be consistent across all components. Did I get that right?
        Hide
        Hadrian Zbarcea added a comment -

        I attached a patch that should give an idea of what I proposed in this issue. There are few more things remaining to be done:

        • add a MappedEndpointConfiguration extends DefaultEndpointConfiguration that takes all the parameters and stores them in a Map.This will be the class used by default by the DefaultEndpoint and will assist in migration and will preserve backwards compatibility
        • validation, the part described by Keith Babo above, which I didn't touch on. It will require an extra dependency in camel-core on javax.validation.
        • toUriFormat is fairly easy, but I need to define a few annotations like @ProviderConfiguration, @ConsumerConfiguration that would mark some parameters as making sense only one one side of the messaging channel.

        Once the above is done we can start tackling components one by one and add better unit testing for supported configurations.

        Please review and comment (or nod). I will not commit attached patch for a few days.

        Show
        Hadrian Zbarcea added a comment - I attached a patch that should give an idea of what I proposed in this issue. There are few more things remaining to be done: add a MappedEndpointConfiguration extends DefaultEndpointConfiguration that takes all the parameters and stores them in a Map.This will be the class used by default by the DefaultEndpoint and will assist in migration and will preserve backwards compatibility validation, the part described by Keith Babo above, which I didn't touch on. It will require an extra dependency in camel-core on javax.validation. toUriFormat is fairly easy, but I need to define a few annotations like @ProviderConfiguration, @ConsumerConfiguration that would mark some parameters as making sense only one one side of the messaging channel. Once the above is done we can start tackling components one by one and add better unit testing for supported configurations. Please review and comment (or nod). I will not commit attached patch for a few days.
        Hide
        Hadrian Zbarcea added a comment -

        The tricky part will be the changes to the Endpoint interface and not break backward compatibility for those who have their own Component implementation. Probably something that keeps the parameters/values in a Map and is always valid. I started to work on a patch and will post it here for review before committing.

        Show
        Hadrian Zbarcea added a comment - The tricky part will be the changes to the Endpoint interface and not break backward compatibility for those who have their own Component implementation. Probably something that keeps the parameters/values in a Map and is always valid. I started to work on a patch and will post it here for review before committing.
        Hide
        Keith Babo added a comment -

        I think this would be a great change on a number of fronts. First, it would allow for a nice separation of the configuration info from the behavior of a component endpoint (e.g. Endpoint, Service, etc.). Second, the fact that it's a JavaBean would mean that the mapping to XML would be relatively straightforward. Perhaps through JAXB annotations or just using the default mapping rules in JAXB without annotations. Finally, validation would go above and beyond what you could do with straight XML schema. We have already gone down the route of defining separate XML schema for Camel endpoints, an exercise I would happily abandon if we could get this type of support up and running.

        I went ahead and played around with a crude example using TimerEndpoint as an example. My guess is the configuration bean would look something like the below. Field-level annotations provide some basic support and the custom constraint @TimeDelayConstraint can be used to perform more advanced validation on the config object as a whole (in this case, verifying that 'time' and 'delay' are not specified together).

        @TimeDelayConstraint
        public class TimerEndpointConfiguration implements EndpointConfiguration {
        
            private Date time;
            @Min(0)
            private long period = 1000;
            @Min(0)
            private long delay;
            private boolean fixedRate;
            private boolean daemon = true;
            private Timer timer;
            @NotNull
            private String timerName;
            
            public String getTimerName() {
                return timerName;
            }
            
            public void setTimerName(String timerName) {
                this.timerName = timerName;
            }
        
            // etc., etc.
        }
        

        If there's interest in moving this forward, I would love to help.

        Show
        Keith Babo added a comment - I think this would be a great change on a number of fronts. First, it would allow for a nice separation of the configuration info from the behavior of a component endpoint (e.g. Endpoint, Service, etc.). Second, the fact that it's a JavaBean would mean that the mapping to XML would be relatively straightforward. Perhaps through JAXB annotations or just using the default mapping rules in JAXB without annotations. Finally, validation would go above and beyond what you could do with straight XML schema. We have already gone down the route of defining separate XML schema for Camel endpoints, an exercise I would happily abandon if we could get this type of support up and running. I went ahead and played around with a crude example using TimerEndpoint as an example. My guess is the configuration bean would look something like the below. Field-level annotations provide some basic support and the custom constraint @TimeDelayConstraint can be used to perform more advanced validation on the config object as a whole (in this case, verifying that 'time' and 'delay' are not specified together). @TimeDelayConstraint public class TimerEndpointConfiguration implements EndpointConfiguration { private Date time; @Min(0) private long period = 1000; @Min(0) private long delay; private boolean fixedRate; private boolean daemon = true ; private Timer timer; @NotNull private String timerName; public String getTimerName() { return timerName; } public void setTimerName( String timerName) { this .timerName = timerName; } // etc., etc. } If there's interest in moving this forward, I would love to help.

          People

          • Assignee:
            Hadrian Zbarcea
            Reporter:
            Hadrian Zbarcea
          • Votes:
            1 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:

              Development