ActiveMQ .Net
  1. ActiveMQ .Net
  2. AMQNET-293

Consumer is not recovered after failover reconnect if connection had been lost in OnMessage (before SendACK)

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.4.0, 1.4.1
    • Fix Version/s: 1.5.0
    • Component/s: NMS
    • Labels:
      None
    • Environment:

      Windows XP - however I believe it's not dependent on the operating system

      Description

      To reproduce the error write the simple WinForm application in C# with listbox (or console app)
      1. Create the connection: failover:(tcp://localhost:61616?keepAlive=true) and start it.
      2. Create the session and the QueueConsumer for queue e.g. "TestQueue" (in default AutoAcknoledged mode)
      3. For queue consumer set the message listener e.g. OnMessage
      4. In OnMessage method do something like Sleep(5000) and then display the received textMessage (by Invoke add the messge to listbox).
      5. By means of localhost:8161/admin create the TestQueue and put to this queue about 20 persistent text messages.
      6. Run the application. You should get on the screen new line every 5 seconds.
      7. Restart the ActiveMQ broker (I'm using 5.3.0.5)
      8. After restarting the broker you stop receiving the messages.
      9. Restart broker again. And you will start getting the messages.

      ----------

      The problem is in NMS.
      Most likely when you restart ActiveMQ broker the client app will be in OnMessage method (just sleeping there for 5 seconds). When those 5 seconds is over then the NMS is trying to SendACK. And this method will not end until failover thread successfully reconnect.. For that time there is a lock on the unconsumedMessages.SynchRoot (see MessageConsumer.Dispatch method). And this fact is painful for another thread which is trying to do the unconsumedMessage.Clear() and needs the locked resource. (this thread is initated in Connection.OnTransportInterrupted() and it wants to call ClearMessagesInProgress on MessageConsumer).
      The worst thing is that the MessageConsumer.ClearMessagesInProgress method cannot call (as waits for locked resource) TransportInterruptionProcessingComplete() which I guess registers consumers which have to be recovered when the connection is back.
      So when the connection is back:
      SendAck completes
      Failover thread DoRecover but does not find our consumer.
      Dispatch method unlocks the unconsumedMessages.SynchRoot
      The working thread registers consumers to be recovered (absolutely too late!!!!)

      1. CropperCapture[41].jpg
        81 kB
        Mark Gellings
      2. CropperCapture[43].jpg
        91 kB
        Mark Gellings
      3. CropperCapture[44].jpg
        78 kB
        Mark Gellings
      4. MessageConsumerDirtyPatch.cs
        3 kB
        Jan Vogelgesang
      5. UnitTestAndPatch.zip
        9 kB
        Jan Vogelgesang

        Activity

        No work has yet been logged on this issue.

          People

          • Assignee:
            Timothy Bish
            Reporter:
            Jan Vogelgesang
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development