Camel
  1. Camel
  2. CAMEL-2206

Addition of 'Sample' Throttling strategy

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.1.0
    • Component/s: camel-core, camel-spring
    • Labels:
      None
    • Patch Info:
      Patch Available

      Description

      I've been using a different kind of sampling based throttling pattern recently and figured others might find it useful. The basic premise is the same as the current throttler, there is a consumer downstream that has specific requirements as to how fast it can process incoming exchanges and a mechanism is required to 'throttle' the exchanges inbound to it. The difference between the current throttler implementation and the sampling throttler is that where the current throttler queues up all the exchanges it receives, the sampling throttler will allow only a single exchange through for a period and 'drop' all others; dropped exchanges being passed to a stop processor to complete them.

      Its been useful for situations where a route receives many semantically equivalent exchanges in a time period, any single one of which could be used to represent them all. For example, say an upstream component generates a stream of warning alarms and sends these to an email processor to inform the operator. An email for each alarm would be overkill instead one every x minutes or so would be preferred. Also it would be undesirable for the throttler to queue up these warning alarms as when the upstream component ceases sending them the emails should cease. Sampling the warnings would pick one for each x minute period, and stop the rest.

      This would be defined via the java dsl as follows

      from("direct:alarm-notifier").sample(60, TimeUnit.SECONDS).to("direct:alarm-emailer")

      or

      from("direct:alarm-notifier").sample().samplePeriod(1).timeUnits(TimeUnit.SECONDS).to("direct:alarm-emailer")

      or with a default of one per second

      from("direct:alarm-notifier").sample().to("direct:alarm-emailer")

      In spring xml it looks like

      <route>
        <from uri="direct:sample" />
        <sample>
          <to uri="mock:result" />
        </sample>
      </route>
      

      or

      <route>
        <from uri="direct:sample" />
        <sample samplePeriod="1" units="seconds">
          <to uri="mock:result" />
        </sample>
      </route>
      

      If this patch is accepted, I'd be happy to update the wiki with the required documentation. As per ususal I've split the patches, one for core and one for spring

      1. sampling-patch-spring.txt
        4 kB
        Stephen Gargan
      2. sampling-patch-core-with-stats.txt
        19 kB
        Stephen Gargan
      3. sampling-patch-core-with-javadoc.txt
        18 kB
        Stephen Gargan
      4. sampling-patch-core.txt
        16 kB
        Stephen Gargan
      5. sampling-patch-core.txt
        18 kB
        Stephen Gargan

        Activity

        Hide
        Claus Ibsen added a comment -

        Closing old resolved issues

        Show
        Claus Ibsen added a comment - Closing old resolved issues
        Hide
        Claus Ibsen added a comment -

        trunk: 883614.

        Thanks a lot for the contribution Stephen.

        I have committed it and create a wiki page
        http://cwiki.apache.org/confluence/display/CAMEL/Sampling

        When you have the ICLA in place you will be able to edit the documentations as well.

        In the mean time feel free to comment/feedback the current wiki page.

        Show
        Claus Ibsen added a comment - trunk: 883614. Thanks a lot for the contribution Stephen. I have committed it and create a wiki page http://cwiki.apache.org/confluence/display/CAMEL/Sampling When you have the ICLA in place you will be able to edit the documentations as well. In the mean time feel free to comment/feedback the current wiki page.
        Hide
        Stephen Gargan added a comment -

        Claus,

        I've made the changed you suggested. It may be marginally more efficient to calculate the stats by maintaining only the total number of samples and the total no of exchanges as the number dropped will always be the total - the no. sampled. I guess if you're running in TRACE mode though the question of efficiency is moot so for clarity I've used accumulators for each.

        If you feel the need to change this or any other changes you see fit, then by all means make them, I do not mind

        I'll get back to you with the ICLA

        Thanks,

        ste

        Show
        Stephen Gargan added a comment - Claus, I've made the changed you suggested. It may be marginally more efficient to calculate the stats by maintaining only the total number of samples and the total no of exchanges as the number dropped will always be the total - the no. sampled. I guess if you're running in TRACE mode though the question of efficiency is moot so for clarity I've used accumulators for each. If you feel the need to change this or any other changes you see fit, then by all means make them, I do not mind I'll get back to you with the ICLA Thanks, ste
        Hide
        Claus Ibsen added a comment -

        I wonder if the DEBUG logging for skipped Exchange should be TRACE level.

        Then it wont spam logs for high throughputs and you run with DEBUG logging.
        And the Exchange that is picked up as sample you could log how many was skipped in that period

        For example the log could be a DEBUG level something like:
        Sampling exchange out of XXX in the last XXX units

        Show
        Claus Ibsen added a comment - I wonder if the DEBUG logging for skipped Exchange should be TRACE level. Then it wont spam logs for high throughputs and you run with DEBUG logging. And the Exchange that is picked up as sample you could log how many was skipped in that period For example the log could be a DEBUG level something like: Sampling exchange out of XXX in the last XXX units
        Hide
        Claus Ibsen added a comment -

        Stephen to edit the wiki pages you need to sign a ICLA

        Read more here
        http://camel.apache.org/how-do-i-edit-the-website.html

        If you dont want to go that route you can create some form of documentation of the sampler and attach it to this JIRA and then I can create and edit the wiki page.

        However by having the ICLA signed you are one step closer to contribute more and becoming a committer.
        http://camel.apache.org/contributing.html
        http://camel.apache.org/how-do-i-become-a-committer.html

        Show
        Claus Ibsen added a comment - Stephen to edit the wiki pages you need to sign a ICLA Read more here http://camel.apache.org/how-do-i-edit-the-website.html If you dont want to go that route you can create some form of documentation of the sampler and attach it to this JIRA and then I can create and edit the wiki page. However by having the ICLA signed you are one step closer to contribute more and becoming a committer. http://camel.apache.org/contributing.html http://camel.apache.org/how-do-i-become-a-committer.html
        Hide
        Stephen Gargan added a comment -

        Whoops sorry about that. I've added the Adapter and updated the javadoc. If you give me write access for the path http://camel.apache.org/sampling.html write up the doc for it.

        Nice one,

        Stephen.

        Show
        Stephen Gargan added a comment - Whoops sorry about that. I've added the Adapter and updated the javadoc. If you give me write access for the path http://camel.apache.org/sampling.html write up the doc for it. Nice one, Stephen.
        Hide
        Stephen Gargan added a comment -

        Complete core patch this time.

        Show
        Stephen Gargan added a comment - Complete core patch this time.
        Hide
        Claus Ibsen added a comment -

        Lets see if we can get it into 2.1

        Show
        Claus Ibsen added a comment - Lets see if we can get it into 2.1
        Hide
        Claus Ibsen added a comment -

        Stephen looks really great.

        I cannot find the class
        import org.apache.camel.builder.xml.TimeUnitAdapter;

        in the patch. Can you attach that file also?

        We would definitely like to add this to the Camel. Thanks for contributing.

        Show
        Claus Ibsen added a comment - Stephen looks really great. I cannot find the class import org.apache.camel.builder.xml.TimeUnitAdapter; in the patch. Can you attach that file also? We would definitely like to add this to the Camel. Thanks for contributing.

          People

          • Assignee:
            Claus Ibsen
            Reporter:
            Stephen Gargan
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development