UIMA
  1. UIMA
  2. UIMA-1902

UIMA AS detection of missing client reply queue is not working leading to client timeouts

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.3AS
    • Fix Version/s: 2.3.1AS
    • Component/s: Async Scaleout
    • Labels:
      None

      Description

      UIMA AS service checks for existence of a reply queue before processing an incoming request. This is done to prevent wasting cycles processing a request if it is known that the client has gone away. Current approach is based on checking broker's queue inventory for presence of a temp reply queue provided in a request msg. The framework code uses JMX connection to a broker to perform the queue lookup. If the lookup fails, the framework concludes that the client has gone away, places the reply queue in a list of destinations known to be dead and throws away the request msg. Any subsequent msg containing the same reply queue is automatically thrown away since the destination is marked as dead. Occasionally, the JMX lookup fails even though (as reported by some users) the client is alive at the time the lookup was done. It seems that perhaps under load JMX fails, reporting false positive result ( queue not present). The failed lookup results in a dropped request msg and ultimately in a client timeout (or possible hang if the client is not using timout).

      Modify the implementation to stop using JMX to perform lookups. Instead, before any processing takes place, send a small msg (ACK) back to the client via JMS. If the send fails, report the problem, drop the request and wait for the next request. Also, dont add failed destination to the list of clients known to be dead. Each request should be subject to an ACK no matter what happened previously.

      The new approach, although generating more jms traffic, has some advantages:

      1) A client is immediately notified when its request has been picked by a service and is ready to be processed.
      2) A client "knows" which service is processing each request
      3) A client can use a different timeout strategy since it knows precisely when its request is ready to be processed.

        Activity

        Hide
        Jerry Cwiklik added a comment -

        Modified UIMA AS service code to send an ACK message via JMS for every input CAS received. The ACK is sent to a reply queue of the client. If the send fails, the CAS is dropped. The code doesnt cache the fact that the reply no longer exists.

        Show
        Jerry Cwiklik added a comment - Modified UIMA AS service code to send an ACK message via JMS for every input CAS received. The ACK is sent to a reply queue of the client. If the send fails, the CAS is dropped. The code doesnt cache the fact that the reply no longer exists.

          People

          • Assignee:
            Jerry Cwiklik
            Reporter:
            Jerry Cwiklik
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development