Uploaded image for project: 'ActiveMQ Artemis'
  1. ActiveMQ Artemis
  2. ARTEMIS-908

AMQP flow control misses unblock during heavy load

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 2.1.0
    • AMQP
    • None
    • Patch

    Description

      I have a benchmarking setup with 1 sender and 1 receiver attached to a broker. The broker is configured with BLOCK policy when queue is full, and I've deliberately set a low global-max-size to trigger this bug.

      While sending messages, I get tons of these in the Artemis log, due to the flow control kicking in:

      10:46:53,673 WARN [org.apache.activemq.artemis.core.server] AMQ222183: Blocking message production on address 'myqueue'; size is currently: 105,456 bytes; max-size-bytes: -1
      10:46:53,674 INFO [org.apache.activemq.artemis.core.server] AMQ221046: Unblocking message production on address 'myqueue'; size is currently: 105,456 bytes; max-size-bytes: -1

      However, usually after running the test for 10-30 seconds, the sender is suddenly never unblocked and I have to restart the sender to send more messages. Every time I attach the sender, I am able to send messages for 10~30 seconds until it stops.

      This happens with different clients, and triggers only if it can send messages really fast.

      After inspecting the flow control logic in the AMQP implementation, I noticed that there is a missing synchronized block when issuing credits in AMQPSessionCallback#offerProducerCredit, which is present in ProtonServerReceiverContext#flow (if sessionSPI is null).

      Adding a synchronized block with the connection lock to the function passed to store.checkMemory() in AMQPSessionCallback#offerProducerCredit (in the same way as in ProtonServerReceiverContext#flow), the sender seems to be unblocked correctly. I've been able to run the sender and receiver for 5 minutes without issues so far.

      I'll submit the patch as a github PR.

      Attachments

        Activity

          People

            jbertram Justin Bertram
            lulf Ulf Lilleengen
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: