Synapse
  1. Synapse
  2. SYNAPSE-168

java.lang.IllegalStateException: I/O reactor has been shut down

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 1.1-RC1
    • Fix Version/s: 1.1.1-QA-B2
    • Component/s: Transports
    • Labels:
      None
    • Environment:
      java 1.5.0_07, Mac OS X 10.4.10, endpoint made with Spring-WS deployed on JBoss 4.2.1

      Description

      Synapse configuration:

      <definitions xmlns="http://ws.apache.org/ns/synapse">

      <in>
      <log level="full"/>

      <send>
      <endpoint>
      <wsdl uri="http://hostname:2006/echo/echo.wsdl" service="EchoService" port="EchoPort">

      <suspendDurationOnFailure>5</suspendDurationOnFailure>
      <timeout>
      <duration>3</duration>
      <action>fault</action>
      </timeout>

      </wsdl>
      </endpoint>
      </send>

      </in>
      <out>
      <send/>
      </out>
      </definitions>

      When simple 'echo' endpoint is deployed and working - everything is OK.
      But when I undeploy it and ask Synapse, I've got series of exception:

      First query:
      2007-11-05 10:06:18,460 [ CUT ] [HttpServerWorker-5] INFO LogMediator To: /soap, WSAction: , SOAPAction: , Direction: request, Envelope: <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:echo="http://www.springframework.org/spring-ws/samples/echo"><soapenv:Body>
      <echo:echoRequest>test123</echo:echoRequest>
      </soapenv:Body></soapenv:Envelope>
      Exception in thread "HttpCoreNIOSender" java.lang.NullPointerException
      at org.apache.synapse.transport.nhttp.ClientHandler.inputReady(ClientHandler.java:225)
      at org.apache.synapse.transport.nhttp.LoggingNHttpClientHandler.inputReady(LoggingNHttpClientHandler.java:113)
      at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:135)
      at org.apache.synapse.transport.nhttp.PlainClientIOEventDispatch.inputReady(PlainClientIOEventDispatch.java:71)
      at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:94)
      at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:189)
      at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:174)
      at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:137)
      at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:69)
      at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:281)
      at java.lang.Thread.run(Thread.java:613)

      Second query:
      2007-11-05 10:20:25,461 [ CUT ] [HttpServerWorker-14] ERROR Axis2Sender Unexpected error during sending message out
      java.lang.IllegalStateException: I/O reactor has been shut down
      at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.connect(DefaultConnectingIOReactor.java:164)
      at org.apache.synapse.transport.nhttp.HttpCoreNIOSender.sendAsyncRequest(HttpCoreNIOSender.java:285)
      at org.apache.synapse.transport.nhttp.HttpCoreNIOSender.invoke(HttpCoreNIOSender.java:208)
      at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:396)
      at org.apache.synapse.core.axis2.DynamicAxisOperation$DynamicOperationClient.send(DynamicAxisOperation.java:192)
      at org.apache.synapse.core.axis2.DynamicAxisOperation$DynamicOperationClient.executeImpl(DynamicAxisOperation.java:176)
      at org.apache.axis2.client.OperationClient.execute(OperationClient.java:163)
      at org.apache.synapse.core.axis2.Axis2FlexibleMEPClient.send(Axis2FlexibleMEPClient.java:250)
      at org.apache.synapse.core.axis2.Axis2Sender.sendOn(Axis2Sender.java:55)
      at org.apache.synapse.core.axis2.Axis2SynapseEnvironment.send(Axis2SynapseEnvironment.java:200)
      at org.apache.synapse.endpoints.WSDLEndpoint.send(WSDLEndpoint.java:140)
      at org.apache.synapse.mediators.builtin.SendMediator.mediate(SendMediator.java:100)
      at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:60)
      at org.apache.synapse.mediators.filters.InMediator.mediate(InMediator.java:60)
      at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:60)
      at org.apache.synapse.mediators.base.SequenceMediator.mediate(SequenceMediator.java:122)
      at org.apache.synapse.core.axis2.Axis2SynapseEnvironment.injectMessage(Axis2SynapseEnvironment.java:154)
      at org.apache.synapse.core.axis2.SynapseMessageReceiver.receive(SynapseMessageReceiver.java:89)
      at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:145)
      at org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:275)
      at org.apache.synapse.transport.nhttp.ServerWorker.processPost(ServerWorker.java:218)
      at org.apache.synapse.transport.nhttp.ServerWorker.run(ServerWorker.java:182)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
      at java.lang.Thread.run(Thread.java:613)

      End every next query ends with this exception.
      Even if I redeploy echo endpoint, Synapse does not process my requests and responses with 0 bytes:

      Mon Nov 05 10:13:59 CET 2007:INFO:Got response for [EchoBinding.echo:Request 1] in 18ms (0 bytes)

      And after a few seconds in logs I have:

      2007-11-05 10:19:06,408 [ CUT ] [Timer-0] INFO LogMediator To: /soap, WSAction: , SOAPAction: , Direction: request, Envelope: <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:echo="http://www.springframework.org/spring-ws/samples/echo"><soapenv:Body>
      <echo:echoRequest>dupa123</echo:echoRequest>
      </soapenv:Body></soapenv:Envelope>

      After restarting Synapse everything backs to normal - so it isn't endpoint's issue.

      1. echo.zip
        6.73 MB
        Maciej Majewski
      2. error_response.txt
        1 kB
        Maciej Majewski
      3. normal_response.txt
        0.6 kB
        Maciej Majewski
      4. query.txt
        0.5 kB
        Maciej Majewski
      5. response_after_permgen.txt
        0.1 kB
        Maciej Majewski

        Activity

        Hide
        Asankha C. Perera added a comment -

        Can you reproduce this in a Linux environment? I am not very familiar with MacOS or have access to a system with it. Looking at the code where the NPE occurred it seems like it could be host OS specific. Also let me know the client HTTP version and if chunking and keepalives are used. If you are sending thousands of requests, you may also want to tune your OS to help Synapse reach its peak levels.

        Show
        Asankha C. Perera added a comment - Can you reproduce this in a Linux environment? I am not very familiar with MacOS or have access to a system with it. Looking at the code where the NPE occurred it seems like it could be host OS specific. Also let me know the client HTTP version and if chunking and keepalives are used. If you are sending thousands of requests, you may also want to tune your OS to help Synapse reach its peak levels.
        Hide
        Asankha C. Perera added a comment -

        This could not be reproduced and I consider this as not a defect - however, more information may prove useful to determine actual cause and solution

        the endpoint definition:
        <wsdl uri="http://hostname:2006/echo/echo.wsdl" service="EchoService" port="EchoPort">

        <suspendDurationOnFailure>5</suspendDurationOnFailure>
        <timeout>
        <duration>3</duration>
        <action>fault</action>
        </timeout>

        </wsdl>

        will not send back an error when the EPR is down, as what is specified is to throw a "fault" if the response arrives later than 3 seconds after the request was sent. If the EPR was found to be offline, it will be considered an error to send to this endpoint during the next 5 seconds after this was found out.

        When Synapse encounters an error as above (e.g. Connection refused, late reply in the above scenario etc) it will execute the onError error handler sequence for the sequence that generates the error. If none can be found the default "fault" sequence would execute (unless defined, this defaults to a simple log of the current message)

        Show
        Asankha C. Perera added a comment - This could not be reproduced and I consider this as not a defect - however, more information may prove useful to determine actual cause and solution the endpoint definition: <wsdl uri="http://hostname:2006/echo/echo.wsdl" service="EchoService" port="EchoPort"> <suspendDurationOnFailure>5</suspendDurationOnFailure> <timeout> <duration>3</duration> <action>fault</action> </timeout> </wsdl> will not send back an error when the EPR is down, as what is specified is to throw a "fault" if the response arrives later than 3 seconds after the request was sent. If the EPR was found to be offline, it will be considered an error to send to this endpoint during the next 5 seconds after this was found out. When Synapse encounters an error as above (e.g. Connection refused, late reply in the above scenario etc) it will execute the onError error handler sequence for the sequence that generates the error. If none can be found the default "fault" sequence would execute (unless defined, this defaults to a simple log of the current message)
        Hide
        Maciej Majewski added a comment - - edited

        Everything works fine, when you test on SimpleStockQuoteService and just shutdown it. This is situation when endpoint do not answer (service is down, connection refused).

        But...

        There is a problem when we undeploy app and application server returns something, e.g. malformed HTML page.

        Please try it with Echo endpoint (will attach .war).

        Sorry but cannot attach it, will try again later from home.

        Show
        Maciej Majewski added a comment - - edited Everything works fine, when you test on SimpleStockQuoteService and just shutdown it. This is situation when endpoint do not answer (service is down, connection refused). But... There is a problem when we undeploy app and application server returns something, e.g. malformed HTML page. Please try it with Echo endpoint (will attach .war). — Sorry but cannot attach it, will try again later from home.
        Hide
        Asankha C. Perera added a comment -

        Seems like a JBoss issue.. if the service is not available what does it return? is it a HTTP 404? or 500? with a HTML message too? Please capture the response from JBoss when the service is undeployed (using TCPMon etc) and attach it for review. I will either need this, or the exact JBoss version and your WAR file to reproduce the scenario

        You may try to Zip the war file and then attach to this JIRA if you have problems

        Show
        Asankha C. Perera added a comment - Seems like a JBoss issue.. if the service is not available what does it return? is it a HTTP 404? or 500? with a HTML message too? Please capture the response from JBoss when the service is undeployed (using TCPMon etc) and attach it for review. I will either need this, or the exact JBoss version and your WAR file to reproduce the scenario You may try to Zip the war file and then attach to this JIRA if you have problems
        Hide
        Maciej Majewski added a comment -

        Attaching query, normal response and 404 error.
        This 404 response causes exceptions.

        Show
        Maciej Majewski added a comment - Attaching query, normal response and 404 error. This 404 response causes exceptions.
        Hide
        Maciej Majewski added a comment - - edited

        And response when JBoss got OutOfMemory / PermGen

        16:25:02,197 ERROR [STDERR] java.lang.OutOfMemoryError: PermGen space

        This also causes exceptions in Synapse.

        Show
        Maciej Majewski added a comment - - edited And response when JBoss got OutOfMemory / PermGen 16:25:02,197 ERROR [STDERR] java.lang.OutOfMemoryError: PermGen space This also causes exceptions in Synapse.
        Hide
        Asankha C. Perera added a comment -

        The problem is that JBoss returns an error message in HTML targeted for a browser, when a Web service encounters a fault or is unavailable. The typical response expected by Synapse in such a scenario is a SOAP fault. We may be able to address this issue as an enhancement over our typical behavior - but this would take some investigation.

        I do not understand what you mean by JBoss running out of memory? As I understand Synapse runs on a separate host than JBoss? but do you mean that if JBoss responds with a HTTP 500 without a body, Synapse encounters an exception? If so the same enhancement I propose could address this issue as well

        Show
        Asankha C. Perera added a comment - The problem is that JBoss returns an error message in HTML targeted for a browser, when a Web service encounters a fault or is unavailable. The typical response expected by Synapse in such a scenario is a SOAP fault. We may be able to address this issue as an enhancement over our typical behavior - but this would take some investigation. I do not understand what you mean by JBoss running out of memory? As I understand Synapse runs on a separate host than JBoss? but do you mean that if JBoss responds with a HTTP 500 without a body, Synapse encounters an exception? If so the same enhancement I propose could address this issue as well
        Hide
        Maciej Majewski added a comment -

        Echo from Spring-WS examples

        Show
        Maciej Majewski added a comment - Echo from Spring-WS examples
        Hide
        Ruwan Linton added a comment -

        changed the fix version

        Show
        Ruwan Linton added a comment - changed the fix version
        Hide
        Asankha C. Perera added a comment -

        Fixed this and SYNAPSE-188 which is almost the same

        Show
        Asankha C. Perera added a comment - Fixed this and SYNAPSE-188 which is almost the same

          People

          • Assignee:
            Asankha C. Perera
            Reporter:
            Maciej Majewski
          • Votes:
            1 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development