[UIMA-2401] BaseUIMAAsynchronousEngine_impl does not protect access to SharedConnection in a multi-instance, multi-threaded scenario. Possible javax.jms.IllegalStateException.

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.1AS
• Fix Version/s: 2.4.0AS
• Component/s: Async Scaleout
• Labels: ibm

Environment

RedHat Enterprise Linux 6.0, Mac OSX Lion 10.7.3

• Assignee: Jaroslaw Cwiklik
• Reporter: Gino Bustelo
• Created: 09/May/12 18:36
• Updated: 30/May/12 16:20
• Resolved: 14/May/12 17:37

Description

We see this problem sporadically when running one instance of BaseUIMAAsynchronousEngine_impl per Thread. This means that each request Thread will instantiate its own instance of BaseUIMAAsynchronousEngine_impl to submit a CAS async. We have noticed in our logs the following exception:

org.apache.uima.adapter.jms.client.BaseUIMAAsynchronousEngine_impl WARN 2012-05-04 00:10:46,710 [BaseUIMAAsynchronousEngine_impl.803] - Top Level Controller Initialization Exception.
javax.jms.IllegalStateException: The Session is closed
at org.apache.activemq.ActiveMQSession.checkClosed(ActiveMQSession.java:722)
at org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:1101)
at org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:1060)
at org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:973)
at org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:946)
at org.apache.uima.adapter.jms.client.BaseUIMAAsynchronousEngine_impl.initializeConsumer(BaseUIMAAsynchronousEngine_impl.java:523)
at org.apache.uima.adapter.jms.client.BaseUIMAAsynchronousEngine_impl.initialize(BaseUIMAAsynchronousEngine_impl.java:687)

Under further inspection, it seems that the BaseUIMAAsynchronousEngine_impl.initialize does not properly protects the access to the SharedConnection object. Given the following scenario:

Thread 1 is in the middle of a call to UimaAsynchronousEngine.initialize.
Thread 2 is finishing up and calling UimaAsynchronousEngine.stop.

Thread 1 can be schedule out just before trying to create a Consumer (BaseUIMAAsynchronousEngine_impl.java:523). Thread 2 will call stop() and due to client count, cause the Connection to get closed. When Thread 1 comes back, the Connection and Session instances that it has reference to are now invalid.

I've attached a test case that forces the exception to prove that there is no proper mutex around access to the SharedConnection between initialize and stop. I believe that the code in question is in BaseUIMAAsynchronousEngine_impl.initialize between lines 682 and 700.

createSharedConnection(brokerURI);
synchronized (connectionMux) {
// Reuse existing JMS connection if available
if (sharedConnection != null)

{ initializeProducer(brokerURI, endpoint, sharedConnection.getConnection()); initializeConsumer(brokerURI, sharedConnection.getConnection()); }

else

{ initializeProducer(brokerURI, endpoint); initializeConsumer(brokerURI); }

// Increment number of client instances. SharedConnection object is a static
// and is used to share a single JMS connection. The connection is closed
// when the last client finishes processing and calls stop().
if (sharedConnection != null)

{ sharedConnection.registerClient(this); }

}

That entire block should be protected with the sharedConnectionSemaphore.