UIMA
  1. UIMA
  2. UIMA-2401

BaseUIMAAsynchronousEngine_impl does not protect access to SharedConnection in a multi-instance, multi-threaded scenario. Possible javax.jms.IllegalStateException.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.3.1AS
    • Fix Version/s: 2.4.0AS
    • Component/s: Async Scaleout
    • Labels:
    • Environment:

      RedHat Enterprise Linux 6.0, Mac OSX Lion 10.7.3

      Description

      We see this problem sporadically when running one instance of BaseUIMAAsynchronousEngine_impl per Thread. This means that each request Thread will instantiate its own instance of BaseUIMAAsynchronousEngine_impl to submit a CAS async. We have noticed in our logs the following exception:

      org.apache.uima.adapter.jms.client.BaseUIMAAsynchronousEngine_impl WARN 2012-05-04 00:10:46,710 [BaseUIMAAsynchronousEngine_impl.803] - Top Level Controller Initialization Exception.
      javax.jms.IllegalStateException: The Session is closed
      at org.apache.activemq.ActiveMQSession.checkClosed(ActiveMQSession.java:722)
      at org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:1101)
      at org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:1060)
      at org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:973)
      at org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:946)
      at org.apache.uima.adapter.jms.client.BaseUIMAAsynchronousEngine_impl.initializeConsumer(BaseUIMAAsynchronousEngine_impl.java:523)
      at org.apache.uima.adapter.jms.client.BaseUIMAAsynchronousEngine_impl.initialize(BaseUIMAAsynchronousEngine_impl.java:687)

      Under further inspection, it seems that the BaseUIMAAsynchronousEngine_impl.initialize does not properly protects the access to the SharedConnection object. Given the following scenario:

      Thread 1 is in the middle of a call to UimaAsynchronousEngine.initialize.
      Thread 2 is finishing up and calling UimaAsynchronousEngine.stop.

      Thread 1 can be schedule out just before trying to create a Consumer (BaseUIMAAsynchronousEngine_impl.java:523). Thread 2 will call stop() and due to client count, cause the Connection to get closed. When Thread 1 comes back, the Connection and Session instances that it has reference to are now invalid.

      I've attached a test case that forces the exception to prove that there is no proper mutex around access to the SharedConnection between initialize and stop. I believe that the code in question is in BaseUIMAAsynchronousEngine_impl.initialize between lines 682 and 700.

      createSharedConnection(brokerURI);
      synchronized (connectionMux) {
      // Reuse existing JMS connection if available
      if (sharedConnection != null)

      { initializeProducer(brokerURI, endpoint, sharedConnection.getConnection()); initializeConsumer(brokerURI, sharedConnection.getConnection()); }

      else

      { initializeProducer(brokerURI, endpoint); initializeConsumer(brokerURI); }

      // Increment number of client instances. SharedConnection object is a static
      // and is used to share a single JMS connection. The connection is closed
      // when the last client finishes processing and calls stop().
      if (sharedConnection != null)

      { sharedConnection.registerClient(this); }

      }

      That entire block should be protected with the sharedConnectionSemaphore.

        Activity

        Hide
        Jerry Cwiklik added a comment -

        In addition to the above mentioned changes, the uima-as client code (BaseUIMAAsynchronousEngine_impl.java) has been modified to change ConnectionMux object scope to static. This change guards the connection object from concurrent access.

        Show
        Jerry Cwiklik added a comment - In addition to the above mentioned changes, the uima-as client code (BaseUIMAAsynchronousEngine_impl.java) has been modified to change ConnectionMux object scope to static. This change guards the connection object from concurrent access.
        Hide
        Jerry Cwiklik added a comment -

        Replicated the problem with supplied test case. Modified BaseUIMAAsynchronousEngine_impl to synchronize access to SharedConnection object during initialization and shutdown of the uima-as client to prevent concurrent access by multiple threads. After the change, the test case completes with no exceptions.

        Show
        Jerry Cwiklik added a comment - Replicated the problem with supplied test case. Modified BaseUIMAAsynchronousEngine_impl to synchronize access to SharedConnection object during initialization and shutdown of the uima-as client to prevent concurrent access by multiple threads. After the change, the test case completes with no exceptions.
        Hide
        Gino Bustelo added a comment -

        Eclipse Project with test case. Refer to README.

        Show
        Gino Bustelo added a comment - Eclipse Project with test case. Refer to README.

          People

          • Assignee:
            Jerry Cwiklik
            Reporter:
            Gino Bustelo
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development