UIMA
  1. UIMA
  2. UIMA-1902

UIMA AS detection of missing client reply queue is not working leading to client timeouts

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.3AS
    • Fix Version/s: 2.3.1AS
    • Component/s: Async Scaleout
    • Labels:
      None

      Description

      UIMA AS service checks for existence of a reply queue before processing an incoming request. This is done to prevent wasting cycles processing a request if it is known that the client has gone away. Current approach is based on checking broker's queue inventory for presence of a temp reply queue provided in a request msg. The framework code uses JMX connection to a broker to perform the queue lookup. If the lookup fails, the framework concludes that the client has gone away, places the reply queue in a list of destinations known to be dead and throws away the request msg. Any subsequent msg containing the same reply queue is automatically thrown away since the destination is marked as dead. Occasionally, the JMX lookup fails even though (as reported by some users) the client is alive at the time the lookup was done. It seems that perhaps under load JMX fails, reporting false positive result ( queue not present). The failed lookup results in a dropped request msg and ultimately in a client timeout (or possible hang if the client is not using timout).

      Modify the implementation to stop using JMX to perform lookups. Instead, before any processing takes place, send a small msg (ACK) back to the client via JMS. If the send fails, report the problem, drop the request and wait for the next request. Also, dont add failed destination to the list of clients known to be dead. Each request should be subject to an ACK no matter what happened previously.

      The new approach, although generating more jms traffic, has some advantages:

      1) A client is immediately notified when its request has been picked by a service and is ready to be processed.
      2) A client "knows" which service is processing each request
      3) A client can use a different timeout strategy since it knows precisely when its request is ready to be processed.

        Activity

        Jerry Cwiklik created issue -
        Jerry Cwiklik made changes -
        Field Original Value New Value
        Summary UIMA AS should send an ACK message to a client for each request UIMA AS detection of missing client reply queue is not working leading to client timeouts
        Jerry Cwiklik made changes -
        Description UIMA AS service checks for existence of a reply queue before processing an incoming request. This is done to prevent wasting cycles processing a request if it is known that the client has gone away. Current approach is based on checking broker's queue inventory for presence of a temp reply queue provided in a request msg. The framework code uses JMX connection to a broker to perform the queue lookup. If the lookup fails, the framework concludes that the client has gone away, places the reply queue in a list of destinations known to be dead and throws away the request msg. Any subsequent msg containing the same reply queue is automatically thrown away since the destination is marked as dead. Occasionally, the JMX lookup fails even though (as reported by some users) the client is alive at the time the lookup was done. It seems that perhaps under load JMX fails apart, reporting false positive result ( queue not present). The failed lookup results in a dropped request msg and ultimately in a client timeout (or possible hang if the client is not using timout).

        Modify the implementation to stop using JMX to perform lookups. Instead, before any processing takes place, send a small msg (ACK) back to the client via JMS. If the send fails, report the problem, drop the request and wait for the next request. Also, dont add failed destination to the list of clients known to be dead. Each request should be subject to an ACK no matter what happened previously.

        The new approach, although generating more jms traffic, has some advantages:

        1) A client is immediately notified when its request has been picked by a service and is ready to be processed.
        2) A client "knows" which service is processing each request
        3) A client can use a different timeout strategy since it knows precisely when its request is ready to be processed.
        UIMA AS service checks for existence of a reply queue before processing an incoming request. This is done to prevent wasting cycles processing a request if it is known that the client has gone away. Current approach is based on checking broker's queue inventory for presence of a temp reply queue provided in a request msg. The framework code uses JMX connection to a broker to perform the queue lookup. If the lookup fails, the framework concludes that the client has gone away, places the reply queue in a list of destinations known to be dead and throws away the request msg. Any subsequent msg containing the same reply queue is automatically thrown away since the destination is marked as dead. Occasionally, the JMX lookup fails even though (as reported by some users) the client is alive at the time the lookup was done. It seems that perhaps under load JMX fails, reporting false positive result ( queue not present). The failed lookup results in a dropped request msg and ultimately in a client timeout (or possible hang if the client is not using timout).

        Modify the implementation to stop using JMX to perform lookups. Instead, before any processing takes place, send a small msg (ACK) back to the client via JMS. If the send fails, report the problem, drop the request and wait for the next request. Also, dont add failed destination to the list of clients known to be dead. Each request should be subject to an ACK no matter what happened previously.

        The new approach, although generating more jms traffic, has some advantages:

        1) A client is immediately notified when its request has been picked by a service and is ready to be processed.
        2) A client "knows" which service is processing each request
        3) A client can use a different timeout strategy since it knows precisely when its request is ready to be processed.
        Jerry Cwiklik made changes -
        Status Open [ 1 ] Closed [ 6 ]
        Fix Version/s 2.3.1AS [ 12315335 ]
        Resolution Fixed [ 1 ]

          People

          • Assignee:
            Jerry Cwiklik
            Reporter:
            Jerry Cwiklik
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development