[SM-1850] Some Send Errors Can Cause Undeployment to Wait Forever

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 3.3
• Component/s: servicemix-camel, servicemix-common
• Assignee: Unassigned
• Reporter: Robert H. Pollack
• Created: 24/Apr/09 19:51
• Updated: 27/Apr/09 20:11
• Resolved: 24/Apr/09 20:45

Description

If a component causes an exception to be thrown in org.apache.servicemix.common.EndpointDeliveryChannel.send(), the exchange is nevertheless entered into a hashmap called knownExchanges. Later, when the component shuts down, the shutdown code in org.apache.servicemix.common.AsyncBaseLifeCycle.prepareShutdown() will wait for this exchange to be processed--but this will never happen. The result is that the JBI container completely stops.

Reproducing this problem is easy in Camel (which is where I discovered it); it requires only a one-line Camel route such as this one:

from ("timer:dmftimer1?period=60000").to ("jbi:service:alpha:beta:gamma");

To reproduce this problem:

1. Create a service assembly containing this route.
2. Copy the service assembly to the hotdeploy directory.
3. After the exception is thrown, attempt to shut down the assembly (the simplest way is to control-C the ServiceMix window). The service assembly will not undeploy and the JBI container will not shut down.

I will attach two zip files to assist in reproducing this error:

1. A pre-built service assembly that you can just copy to hotdeploy.
2. Source code and a build script to assist experimentation.
   (The build script is an Ant file that requires you to point the property camel-dir at a directory that contains the Camel core jar.

The relevant code in EndpointDeliveryChannel is this:

    public void send(MessageExchange exchange) throws MessagingException {
        prepareExchange(exchange);
        handleExchange(exchange, exchange.getStatus() == ExchangeStatus.ACTIVE);
        channel.send(exchange);
    }

The call to handleExchange() enters the exchange into the hashmap. But if channel.send() throws an exception, the exchange is never removed from the hashmap.

The relevant code in AsyncBaseLifeCycle is this:

    public void prepareShutdown(Endpoint endpoint) throws InterruptedException {
        Set<String> exchanges = getKnownExchanges(endpoint);
        synchronized (exchanges) {
            if (!exchanges.isEmpty()) {
                exchanges.wait();
            }
        }
    }

I do not know if all send problems cause this behavior. I discovered it by running a Camel application that attempts to send a message to a non-existent JBI endpoint. When the Camel program does this, I get a stack trace that (in part) looks like this:

   ...
   Caused by: javax.jbi.messaging.MessagingException: Could not find route for exchange: InOnly[
     id: ID:10.6.10.123-120d97c610d-3:0
     status: Active
     role: provider
     service: {alpha:beta}gamma
     in: null
   ] for service: {alpha:beta}gamma and interface: null
           at org.apache.servicemix.jbi.nmr.DefaultBroker.sendExchangePacket(DefaultBroker.java:297)
           at org.apache.servicemix.jbi.security.SecuredBroker.sendExchangePacket(SecuredBroker.java:88)
           at org.apache.servicemix.jbi.container.JBIContainer.sendExchange(JBIContainer.java:894)
           at org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.doSend(DeliveryChannelImpl.java:395)
           at org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.send(DeliveryChannelImpl.java:431)
           at org.apache.servicemix.common.EndpointDeliveryChannel.send(EndpointDeliveryChannel.java:88)
           at org.apache.servicemix.common.endpoints.SimpleEndpoint.send(SimpleEndpoint.java:66)
           at org.apache.servicemix.camel.CamelConsumerEndpoint.process(CamelConsumerEndpoint.java:89)
           ... 13 more

Later, when attempting to shut down, a thread dump includes the following:

   "Thread-16" daemon prio=6 tid=0x2bb34400 nid=0x1398 in Object.wait() [0x2ddff000..0x2ddffc94]
      java.lang.Thread.State: WAITING (on object monitor)
           at java.lang.Object.wait(Native Method)
           - waiting on <0x25aed9f8> (a java.util.HashSet)
           at java.lang.Object.wait(Object.java:485)
           at org.apache.servicemix.common.AsyncBaseLifeCycle.prepareShutdown(AsyncBaseLifeCycle.java:657)
           - locked <0x25aed9f8> (a java.util.HashSet)
           at org.apache.servicemix.common.DefaultServiceUnit.removeEndpoint(DefaultServiceUnit.java:205)
           - locked <0x0826a048> (a org.apache.servicemix.common.xbean.XBeanServiceUnit)
           at org.apache.servicemix.common.DefaultComponent.removeEndpoint(DefaultComponent.java:301)
           - locked <0x08269958> (a org.apache.servicemix.camel.CamelJbiComponent)
           at org.apache.servicemix.camel.JbiEndpoint$JbiProducer.stop(JbiEndpoint.java:82) ...

My proposed fix is to change the code in EndpointDeliveryChannel to a try-catch block, as shown below. But I defer to those who know ServiceMix better than I to say whether this needs to be more sophisticated. My change, which I have tested and which works fine for me, is as follows:

    public void send(MessageExchange exchange) throws MessagingException {
        prepareExchange(exchange);
        handleExchange(exchange, exchange.getStatus() == ExchangeStatus.ACTIVE);
        try {
           channel.send(exchange);
        } catch (MessagingException me) {
           // If something goes wrong with the send, remove this exchange
           // from the list of known exchanges or else the caller will
           // never be able to undeploy
           // (see AsyncBaseLifeCycle.prepareShutdown()).

           handleExchange (exchange, false);
           throw me;
        } catch (Throwable t) {
           handleExchange (exchange, false);
           throw new MessagingException (t.getMessage(), t);
        }
    }