Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.15.0
    • Component/s: camel-core, eip
    • Labels:
      None
    • Estimated Complexity:
      Unknown

      Description

      Look at add the circuit breaker EIP to the Camel DSL.
      http://davybrion.com/blog/2008/05/the-circuit-breaker/

      Would need some thoughts for that though. Either as an explicit in the DSL. Or as a interceptor for sending to an endpoint. As explicit its a kind to the load balancer (in fact it may be extended upon that). Either the LB selects the intended target, or it select the breaker, which rejects executing the message.

      1. CAMEL-5539-output-82ef800.txt
        3 kB
        Gregor Zurowski
      2. CAMEL-5539-output.txt
        3 kB
        Gregor Zurowski

        Issue Links

          Activity

          Hide
          davsclaus Claus Ibsen added a comment -

          Lets close this ticket as we have a circuit breaker eip released for a while.

          New ideas and whatnot, log a new ticket

          Show
          davsclaus Claus Ibsen added a comment - Lets close this ticket as we have a circuit breaker eip released for a while. New ideas and whatnot, log a new ticket
          Hide
          raj3sh1@gmail.com Rajesh Chitharanjan added a comment -

          Hi Andrew,
          Did your implementation work out? We are planning to use Hystrix within Camel and I was wondering if you are planning to OSS your implementation.

          Show
          raj3sh1@gmail.com Rajesh Chitharanjan added a comment - Hi Andrew, Did your implementation work out? We are planning to use Hystrix within Camel and I was wondering if you are planning to OSS your implementation.
          Hide
          yvesdm Yves De Moor added a comment -

          Simple, quick and most flexible solution could be :

          Put the number of failure in an exchange property.
          Put circuit state in an exchange property (2 status : Open / closed, as halfOpen do the same as closed in term of business)
          use a Content Based Router to perform something like

          from("direct:a")
          .loadBalance()//
          .circuitBreaker(3, 3000L, MyCustomException.class)
          .choice()
          .when(property(Exchange.CircuitBreaker_state).isEqualTo("closed"))
          .to("direct:normalMode")
          .when(property(Exchange.CircuitBreaker_state).isEqualTo("opened"))
          .to("direct:degradedMode")
          .end()

          from("direct:degradedMode")
          .choice()
          .when(property(Exchange.CircuitBreaker_errorCount).isEqualTo("1")) //First error encountered, send an alert to monitoring
          .to("direct:monitorging_alert_degradedMode")
          .end()

          I also see the current solution is not enough robust.
          when halfOpen is reached, (System.currentTimeMillis() - lastFailure < halfOpenAfter) there is no restriction on the number of calls (threads) that will perform a retry.

          When retry timeout is reached, we should let only 1 thread go in "halfOpen" mode, all other should be kept in "open" state until an halfOpen succeed.
          (will create new issue for this)

          Yves.

          Show
          yvesdm Yves De Moor added a comment - Simple, quick and most flexible solution could be : Put the number of failure in an exchange property. Put circuit state in an exchange property (2 status : Open / closed, as halfOpen do the same as closed in term of business) use a Content Based Router to perform something like from("direct:a") .loadBalance()// .circuitBreaker(3, 3000L, MyCustomException.class) .choice() .when(property(Exchange.CircuitBreaker_state).isEqualTo("closed")) .to("direct:normalMode") .when(property(Exchange.CircuitBreaker_state).isEqualTo("opened")) .to("direct:degradedMode") .end() from("direct:degradedMode") .choice() .when(property(Exchange.CircuitBreaker_errorCount).isEqualTo("1")) //First error encountered, send an alert to monitoring .to("direct:monitorging_alert_degradedMode") .end() I also see the current solution is not enough robust. when halfOpen is reached, (System.currentTimeMillis() - lastFailure < halfOpenAfter) there is no restriction on the number of calls (threads) that will perform a retry. When retry timeout is reached, we should let only 1 thread go in "halfOpen" mode, all other should be kept in "open" state until an halfOpen succeed. (will create new issue for this) Yves.
          Hide
          bibryam Bilgin Ibryam added a comment -

          Sounds like a good improvements to add. Feel free to create issues and do PRs

          Show
          bibryam Bilgin Ibryam added a comment - Sounds like a good improvements to add. Feel free to create issues and do PRs
          Hide
          yvesdm Yves De Moor added a comment -

          I would suggest two changes in the current implementation :

          1. Ability to call a route when the the circuit is open and not only when it is closed (to allow for instance to go further with a degraded mode, or to return a functional error, ...). The source code currently add a RejectedExecutionException to the exchange.
          I don't know if this is a camel expected behaviour as it is usually thrown to reject the execution when camel is shutting down.

          2. If change 1 is accepted, we could set the number of failure in the exchange so we could for instance react the first time the circuit is open (write to the log, send an alert, or whatever else)

          What are your opinion ?

          Show
          yvesdm Yves De Moor added a comment - I would suggest two changes in the current implementation : 1. Ability to call a route when the the circuit is open and not only when it is closed (to allow for instance to go further with a degraded mode, or to return a functional error, ...). The source code currently add a RejectedExecutionException to the exchange. I don't know if this is a camel expected behaviour as it is usually thrown to reject the execution when camel is shutting down. 2. If change 1 is accepted, we could set the number of failure in the exchange so we could for instance react the first time the circuit is open (write to the log, send an alert, or whatever else) What are your opinion ?
          Hide
          bibryam Bilgin Ibryam added a comment -

          My implementation is completed but I'll leave this issue open in case other implementation as mentioned above are to be added.

          Show
          bibryam Bilgin Ibryam added a comment - My implementation is completed but I'll leave this issue open in case other implementation as mentioned above are to be added.
          Hide
          andrew.harmel.law Andrew Harmel-Law added a comment -

          Interesting to see this. We'll have to try it out. I mentioned above that we put in Hystrix. To get it working nicely with everything we've been working on an Archaius-spring-camel adapter (Archaius is what Hystrix and other Netflix things uses for config). We're going to OSS it once it's clean and has been used a bit internally. Perhaps we could wrap it in a LoadBalancer too to make things easier for others.

          Show
          andrew.harmel.law Andrew Harmel-Law added a comment - Interesting to see this. We'll have to try it out. I mentioned above that we put in Hystrix. To get it working nicely with everything we've been working on an Archaius-spring-camel adapter (Archaius is what Hystrix and other Netflix things uses for config). We're going to OSS it once it's clean and has been used a bit internally. Perhaps we could wrap it in a LoadBalancer too to make things easier for others.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/camel/pull/131

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/camel/pull/131
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user gzurowski opened a pull request:

          https://github.com/apache/camel/pull/131

          CAMEL-5539: Fix compilation problems

          Fix typo as requested by Bilgin Ibryam <bibryam@apache.org>.

          Signed-off-by: Gregor Zurowski <gregor@zurowski.org>

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/gzurowski/camel CAMEL-5539-FixBuild

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/camel/pull/131.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #131


          commit 1949235f1ae94bbd3fc8e7d7f7b6bb1e854ecae1
          Author: Gregor Zurowski <gregor@zurowski.org>
          Date: 2014-04-07T21:37:28Z

          CAMEL-5539: Fix compilation problems

          Fix typo as requested by Bilgin Ibryam <bibryam@apache.org>.

          Signed-off-by: Gregor Zurowski <gregor@zurowski.org>


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user gzurowski opened a pull request: https://github.com/apache/camel/pull/131 CAMEL-5539 : Fix compilation problems Fix typo as requested by Bilgin Ibryam <bibryam@apache.org>. Signed-off-by: Gregor Zurowski <gregor@zurowski.org> You can merge this pull request into a Git repository by running: $ git pull https://github.com/gzurowski/camel CAMEL-5539 -FixBuild Alternatively you can review and apply these changes as the patch at: https://github.com/apache/camel/pull/131.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #131 commit 1949235f1ae94bbd3fc8e7d7f7b6bb1e854ecae1 Author: Gregor Zurowski <gregor@zurowski.org> Date: 2014-04-07T21:37:28Z CAMEL-5539 : Fix compilation problems Fix typo as requested by Bilgin Ibryam <bibryam@apache.org>. Signed-off-by: Gregor Zurowski <gregor@zurowski.org>
          Hide
          gzurowski Gregor Zurowski added a comment -

          @Bilgin Ibryam: There are still problems building the project:

          ==================================
          [...]
          [INFO] — scala-maven-plugin:3.1.6:compile (default) @ camel-scala —
          [INFO] C:\var\git\forks\camel\components\camel-scala\src\main\scala:-1: info: co
          mpiling
          [INFO] Compiling 43 source files to C:\var\git\forks\camel\components\camel-scal
          a\target\classes at 1396785664444
          [ERROR] C:\var\git\forks\camel\components\camel-scala\src\main\scala\org\apache\
          camel\scala\dsl\SLoadBalanceDefinition.scala:29: error: not found: value classes
          b
          [ERROR] = wrap(target.circuitBreaker(threshold, halfOpenAfter, classesb: _*))
          [ERROR] ^
          [ERROR] one error found
          [...]
          ==================================

          See the full build output in the attachment: https://issues.apache.org/jira/secure/attachment/12638908/CAMEL-5539-output-82ef800.txt

          Show
          gzurowski Gregor Zurowski added a comment - @ Bilgin Ibryam : There are still problems building the project: ================================== [...] [INFO] — scala-maven-plugin:3.1.6:compile (default) @ camel-scala — [INFO] C:\var\git\forks\camel\components\camel-scala\src\main\scala:-1: info: co mpiling [INFO] Compiling 43 source files to C:\var\git\forks\camel\components\camel-scal a\target\classes at 1396785664444 [ERROR] C:\var\git\forks\camel\components\camel-scala\src\main\scala\org\apache\ camel\scala\dsl\SLoadBalanceDefinition.scala:29: error: not found: value classes b [ERROR] = wrap(target.circuitBreaker(threshold, halfOpenAfter, classesb: _*)) [ERROR] ^ [ERROR] one error found [...] ================================== See the full build output in the attachment: https://issues.apache.org/jira/secure/attachment/12638908/CAMEL-5539-output-82ef800.txt
          Hide
          bibryam Bilgin Ibryam added a comment -

          Sorry for that, it is fixed now.

          Show
          bibryam Bilgin Ibryam added a comment - Sorry for that, it is fixed now.
          Hide
          gzurowski Gregor Zurowski added a comment - - edited

          @Bilgin Ibryam: It seems that your recent changes break the build:

          ==================================
          [INFO] Scanning for projects...
          [INFO]
          [INFO] ------------------------------------------------------------------------
          [INFO] Building Camel :: Scala 2.14-SNAPSHOT
          [INFO] ------------------------------------------------------------------------
          [...]
          [INFO] — maven-compiler-plugin:2.5.1:compile (default-compile) @ camel-scala -

          [INFO] Nothing to compile - all classes are up to date
          [INFO]
          [INFO] — scala-maven-plugin:3.1.6:compile (default) @ camel-scala —
          [INFO] C:\var\git\forks\camel\components\camel-scala\src\main\scala:-1: info: co
          mpiling
          [INFO] Compiling 43 source files to C:\var\git\forks\camel\components\camel-scal
          a\target\classes at 1396749824710
          [ERROR] C:\var\git\forks\camel\components\camel-scala\src\main\scala\org\apache\
          camel\scala\dsl\SLoadBalanceDefinition.scala:29: error: type mismatch;
          [ERROR] found : Seq[Class[_]]
          [ERROR] required: Class[_]
          [ERROR] = wrap(target.circuitBreaker(threshold, halfOpenAfter, exceptions))
          [ERROR] ^
          [ERROR] one error found
          [INFO] ------------------------------------------------------------------------
          [INFO] BUILD FAILURE
          [INFO] ------------------------------------------------------------------------
          [INFO] Total time: 28.319s
          [INFO] Finished at: Sat Apr 05 22:04:01 EDT 2014
          [INFO] Final Memory: 20M/303M
          [INFO] ------------------------------------------------------------------------
          [...]
          ==================================

          See full build output attached: https://issues.apache.org/jira/secure/attachment/12638893/CAMEL-5539-output.txt.

          Show
          gzurowski Gregor Zurowski added a comment - - edited @ Bilgin Ibryam : It seems that your recent changes break the build: ================================== [INFO] Scanning for projects... [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building Camel :: Scala 2.14-SNAPSHOT [INFO] ------------------------------------------------------------------------ [...] [INFO] — maven-compiler-plugin:2.5.1:compile (default-compile) @ camel-scala - – [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] — scala-maven-plugin:3.1.6:compile (default) @ camel-scala — [INFO] C:\var\git\forks\camel\components\camel-scala\src\main\scala:-1: info: co mpiling [INFO] Compiling 43 source files to C:\var\git\forks\camel\components\camel-scal a\target\classes at 1396749824710 [ERROR] C:\var\git\forks\camel\components\camel-scala\src\main\scala\org\apache\ camel\scala\dsl\SLoadBalanceDefinition.scala:29: error: type mismatch; [ERROR] found : Seq[Class [_] ] [ERROR] required: Class [_] [ERROR] = wrap(target.circuitBreaker(threshold, halfOpenAfter, exceptions)) [ERROR] ^ [ERROR] one error found [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 28.319s [INFO] Finished at: Sat Apr 05 22:04:01 EDT 2014 [INFO] Final Memory: 20M/303M [INFO] ------------------------------------------------------------------------ [...] ================================== See full build output attached: https://issues.apache.org/jira/secure/attachment/12638893/CAMEL-5539-output.txt .
          Hide
          bibryam Bilgin Ibryam added a comment -

          I've implemented a simple version of circuit breaker pattern a load balancer policy. Let me know what you think.

          Show
          bibryam Bilgin Ibryam added a comment - I've implemented a simple version of circuit breaker pattern a load balancer policy. Let me know what you think.
          Hide
          andrew.harmel.law Andrew Harmel-Law added a comment -

          Thanks guys. We put in Hystrix. Ludicrously simple, and works a charm (we had some spring bean service activators which helped).

          Show
          andrew.harmel.law Andrew Harmel-Law added a comment - Thanks guys. We put in Hystrix. Ludicrously simple, and works a charm (we had some spring bean service activators which helped).
          Hide
          ewheath Eric Heath added a comment -

          These guys are giving it a go as well: https://jira.springsource.org/browse/INT-2250
          Nothing like some friendly competition I guess...

          Show
          ewheath Eric Heath added a comment - These guys are giving it a go as well: https://jira.springsource.org/browse/INT-2250 Nothing like some friendly competition I guess...
          Hide
          ewheath Eric Heath added a comment - - edited

          Andrew, if you haven't seen https://github.com/Netflix/Hystrix yet, do take a look. The netflix guys have really amazing stuff going on.

          You'll find more treats at https://github.com/monitoringsucks as well.

          I would also love to see first class support for circuit breakers in camel. It would be dreamy to configure CBs directly into your routes.

          Show
          ewheath Eric Heath added a comment - - edited Andrew, if you haven't seen https://github.com/Netflix/Hystrix yet, do take a look. The netflix guys have really amazing stuff going on. You'll find more treats at https://github.com/monitoringsucks as well. I would also love to see first class support for circuit breakers in camel. It would be dreamy to configure CBs directly into your routes.
          Hide
          andrew.harmel.law Andrew Harmel-Law added a comment -

          Cheers David. We'll take a look (already got the book, but the src sounds promising)

          Show
          andrew.harmel.law Andrew Harmel-Law added a comment - Cheers David. We'll take a look (already got the book, but the src sounds promising)
          Show
          davidkarlsen@gmail.com David J. M. Karlsen added a comment - Some additional resources: https://code.google.com/p/kite-lib/wiki/CircuitBreaker http://dev.hubspot.com/blog/bid/64543/Building-a-Robust-System-Using-the-Circuit-Breaker-Pattern http://www.amazon.com/Release-It-Production-Ready-Pragmatic-Programmers/dp/0978739213
          Hide
          andrew.harmel.law Andrew Harmel-Law added a comment -

          We're keen to have this functionality. We're running 2.11.1, but as of now, this isn't available. Is there a Work-in-Progress version in a milestone that we can test against for now? We're also looking to wire this into our monitoring infrastructure (StatsD-based). Any update would be much appreciated. If we can avoid writing our own code, that'd be great.

          Show
          andrew.harmel.law Andrew Harmel-Law added a comment - We're keen to have this functionality. We're running 2.11.1, but as of now, this isn't available. Is there a Work-in-Progress version in a milestone that we can test against for now? We're also looking to wire this into our monitoring infrastructure (StatsD-based). Any update would be much appreciated. If we can avoid writing our own code, that'd be great.
          Hide
          iocanel Ioannis Canellos added a comment -

          A must have!

          Show
          iocanel Ioannis Canellos added a comment - A must have!
          Hide
          raulvk Raúl Kripalani added a comment -

          A few initial thoughts on the design of this new EIP:

          • The EIP should wrap around an endpoint or a processor. It's important to cover Processors too because frequently the access external resources (DBs or callouts to partnerlinks).
          • By default, it'll be sensitive to all exceptions, but the user can limit the scope by specifying what exceptions to cover, using the <exception /> DSL.
          • Possible transitions: Closed => Open, Open => HalfOpen, HalfOpen => Open, HalfOpen => Closed. The user may configure actions for each transition:
            • throw an exception, applicable to Closed => Open, Open => HalfOpen
            • invoke another endpoint (e.g. to notify that a transition has happened), or to take an alternative route, applicable to all transitions
          • As David suggests, the algorithm must be pluggable; the strategy interface must be notified twice: before and after an exchange is processed.

          Two options spring to mind with respect to the interaction between the Strategy and the EIP itself:

          1. The strategy gets a control class (the circuit breaker itself) via a setter. It must use this control class to signal when a transition occurs. 'Before' and 'after' callbacks just to do accountancy. When thresholds are surpassed, the appropriate method in the circuit breaker must be called to signal the transition.
          2. 'Before' and 'after' methods return transition enumeration elements to the Circuit Breaker control class, and the Circuit Breaker decides what to do depending on the transition taken. A possible transition is "None".
            • E.g.: the 'after' method returning Transition.ClosedToOpen means that the failure threshold was exceeded. From then on, all 'before' invocations would return Transition.None and the EIP will refrain from sending to the endpoint/processor. When state changes to HalfOpen, the EIP will continue dispatching to the endpoint. If the endpoint fails again, the 'after' method should see an Exception in the Exchange and return Transition.Open again, and so the story goes.

          Please feel free to comment to provide your valuable proposals

          Show
          raulvk Raúl Kripalani added a comment - A few initial thoughts on the design of this new EIP: The EIP should wrap around an endpoint or a processor. It's important to cover Processors too because frequently the access external resources (DBs or callouts to partnerlinks). By default, it'll be sensitive to all exceptions, but the user can limit the scope by specifying what exceptions to cover, using the <exception /> DSL. Possible transitions: Closed => Open, Open => HalfOpen, HalfOpen => Open, HalfOpen => Closed. The user may configure actions for each transition: throw an exception, applicable to Closed => Open, Open => HalfOpen invoke another endpoint (e.g. to notify that a transition has happened), or to take an alternative route, applicable to all transitions As David suggests, the algorithm must be pluggable; the strategy interface must be notified twice: before and after an exchange is processed. Two options spring to mind with respect to the interaction between the Strategy and the EIP itself: The strategy gets a control class (the circuit breaker itself) via a setter. It must use this control class to signal when a transition occurs. 'Before' and 'after' callbacks just to do accountancy. When thresholds are surpassed, the appropriate method in the circuit breaker must be called to signal the transition. 'Before' and 'after' methods return transition enumeration elements to the Circuit Breaker control class, and the Circuit Breaker decides what to do depending on the transition taken. A possible transition is "None". E.g.: the 'after' method returning Transition.ClosedToOpen means that the failure threshold was exceeded. From then on, all 'before' invocations would return Transition.None and the EIP will refrain from sending to the endpoint/processor. When state changes to HalfOpen, the EIP will continue dispatching to the endpoint. If the endpoint fails again, the 'after' method should see an Exception in the Exchange and return Transition.Open again, and so the story goes. Please feel free to comment to provide your valuable proposals
          Hide
          raulvk Raúl Kripalani added a comment -

          I can see this pattern being extremely useful in real life. I don't think there are any ESBs out there (commercial or open source) supporting it. Let's try and add it in 2.11. Changing fix version and assigning to myself.

          Show
          raulvk Raúl Kripalani added a comment - I can see this pattern being extremely useful in real life. I don't think there are any ESBs out there (commercial or open source) supporting it. Let's try and add it in 2.11. Changing fix version and assigning to myself.
          Hide
          davidkarlsen@gmail.com David J. M. Karlsen added a comment -

          Good idea.
          We've already implemented this as an aspect around some inhouse code.
          I would like to add that the actual circuitbreaking would be good to have implemented as a strategy interface, and thus be pluggable.
          in our code we use other criterias for rejecting execution:
          1) that mean time M for processing the last N calls has passed a given threshold X
          2) that more than N current executions are inflight, thus the N+1 is rejected

          Show
          davidkarlsen@gmail.com David J. M. Karlsen added a comment - Good idea. We've already implemented this as an aspect around some inhouse code. I would like to add that the actual circuitbreaking would be good to have implemented as a strategy interface, and thus be pluggable. in our code we use other criterias for rejecting execution: 1) that mean time M for processing the last N calls has passed a given threshold X 2) that more than N current executions are inflight, thus the N+1 is rejected

            People

            • Assignee:
              raulvk Raúl Kripalani
              Reporter:
              davsclaus Claus Ibsen
            • Votes:
              7 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development