Uploaded image for project: 'ActiveMQ Artemis'
  1. ActiveMQ Artemis
  2. ARTEMIS-2560

Duplicate messages created by cluster lead to OOME crash

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 2.10.1
    • 2.11.0
    • AMQP
    • None

    Description

      Summary: When using two node cluster with very simple configuration (see attached broker.xml files), duplicate messages are generated to queue when Python client is used to consume messages one by one from alternating nodes. Duplicate messages are generated until OutOfMemoryException crashes the broker.

      Detailed description how to produce this problem:

      1. Create two node cluster using attached broker.xml files. Node names are server0 and server1 for the rest of this description.
      2. Produce 100 messages to a queue defined in the address configuration inside broker.xml file on node server0. See attached Python producer. Produced messages have identical content. Command to produce the messages using the attached Python producer: "python python-qpid-producer.py -u $username -p $password -H server0 -a exampleQueue -m TestMessageFooBar -A 100"
      3. Consume one message from server1 using attached Python consumer. Cluster will balance the messages to server1 and total amount of messages is decreased by one. After one message is consumed, session and connection are closed. Command to consume a message using attached Python consumer: "python python-qpid-consumer.py -u $username -p $password -H server1 -a exampleQueue"
      4. Consume one message from server0 using attached Python consumer. Cluster will balance the messages to server0 and total amount of messages is decreased by one. After one message is consumed, session and connection are closed. Command to consume a message using attached Python consumer: "python python-qpid-consumer.py -u $username -p $password -H server0 -a exampleQueue"
      5. Consume one message from server1 using attached Python consumer. Cluster will balance the messages to server1 but this time total amount of messages is increased radically.
      6. If consuming of messages is continued in the manner described above (one message from one node and then one message from another node), more messages continue to appear into the queue until broker runs out of memory and crashes.

      Technical details considering Python test described above:

      • Apache ActiveMQ Artemis 2.10.1 on RHEL 7.7 64bit
      • OpenJDK 11.0.5
      • Python 3.4.10
      • Apache Qpid Proton 0.29.0 installed via PIP

      In addition to above, following different variations have been tested. Problem still occurs with all these variations:

      If this is not a bug, I would be very happy for any solution for this problem, wheter it is pointing out some mistake in the configuration or in the consumer, explaining if this is a designed feature or some other explanation.

      Attachments

        1. server1-broker.xml
          3 kB
          Mikko Niemi
        2. server0-broker.xml
          3 kB
          Mikko Niemi
        3. python-stomp-consumer.py
          2 kB
          Mikko Niemi
        4. python-qpid-producer.py
          3 kB
          Mikko Niemi
        5. python-qpid-consumer.py
          2 kB
          Mikko Niemi

        Issue Links

          Activity

            People

              Unassigned Unassigned
              mniemi Mikko Niemi
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h 50m
                  1h 50m