Qpid
  1. Qpid
  2. QPID-4046

Improve broker's performance by rate limiting queue cleanup.

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Trivial Trivial
    • Resolution: Fixed
    • Affects Version/s: 0.16
    • Fix Version/s: 0.17
    • Component/s: C++ Broker
    • Labels:
      None

      Description

      The cleanup of dequeued messages is done on the receive path with the Queue's messageLock held (has to be held). When multiple consumers with large ack windows are sharing the queue, a large backlog of messages can build up. Performance of the consumer can be improved by rate limiting this queue cleanup.

      1. patch.txt
        2 kB
        Ken Giusti

        Activity

        Hide
        Ken Giusti added a comment -

        Proposed fix.

        Show
        Ken Giusti added a comment - Proposed fix.
        Hide
        Ken Giusti added a comment -

        Performance of queue fill followed by queue drain, multiple clients, prior to fix:

        [kgiusti@xxx Test1.5]$ ./Test1-Setup.sh; ./Test1-Sender.sh; ./Test1-Receiver.sh
        + qpid-config -b 127.0.0.1:8888 add queue inQ1 --max-queue-size=12000000000 --max-queue-count=4000000 --flow-stop-size=0 --flow-stop-count=0
        + numactl --cpunodebind 6 qpid-send -b 127.0.0.1:8888 -a inQ1 -m 1000000 --content-size 300 --capacity 2000 --report-total --sequence no --timestamp no
        + numactl --cpunodebind 5 qpid-send -b 127.0.0.1:8888 -a inQ1 -m 1000000 --content-size 300 --capacity 2000 --report-total --sequence no --timestamp no
        + numactl --cpunodebind 4 qpid-send -b 127.0.0.1:8888 -a inQ1 -m 1000000 --content-size 300 --capacity 2000 --report-total --sequence no --timestamp no
        + wait
        + numactl --cpunodebind 3 qpid-send -b 127.0.0.1:8888 -a inQ1 -m 1000000 --content-size 300 --capacity 2000 --report-total --sequence no --timestamp no
        tp(m/s)
        37135
        tp(m/s)
        37044
        tp(m/s)
        37010
        tp(m/s)
        36945
        + numactl --cpunodebind 5 qpid-receive -b 127.0.0.1:8888 -a inQ1 -f -m 1000000 --capacity 2000 --ack-frequency 1000 --print-content no --report-total
        + numactl --cpunodebind 4 qpid-receive -b 127.0.0.1:8888 -a inQ1 -f -m 1000000 --capacity 2000 --ack-frequency 1000 --print-content no --report-total
        + numactl --cpunodebind 3 qpid-receive -b 127.0.0.1:8888 -a inQ1 -f -m 1000000 --capacity 2000 --ack-frequency 1000 --print-content no --report-total
        + numactl --cpunodebind 6 qpid-receive -b 127.0.0.1:8888 -a inQ1 -f -m 1000000 --capacity 2000 --ack-frequency 1000 --print-content no --report-total
        + wait
        tp(m/s) l-min l-max l-avg
        36776
        tp(m/s) l-min l-max l-avg
        35813
        tp(m/s) l-min l-max l-avg
        34623
        tp(m/s) l-min l-max l-avg
        34519

        Same test, with patch to rate limit queue cleanup:

        [kgiusti@xxxx Test1.5]$ ./Test1-Setup.sh; ./Test1-Sender.sh; ./Test1-Receiver.sh
        + qpid-config -b 127.0.0.1:8888 add queue inQ1 --max-queue-size=12000000000 --max-queue-count=4000000 --flow-stop-size=0 --flow-stop-count=0
        + numactl --cpunodebind 6 qpid-send -b 127.0.0.1:8888 -a inQ1 -m 1000000 --content-size 300 --capacity 2000 --report-total --sequence no --timestamp no
        + numactl --cpunodebind 5 qpid-send -b 127.0.0.1:8888 -a inQ1 -m 1000000 --content-size 300 --capacity 2000 --report-total --sequence no --timestamp no
        + numactl --cpunodebind 4 qpid-send -b 127.0.0.1:8888 -a inQ1 -m 1000000 --content-size 300 --capacity 2000 --report-total --sequence no --timestamp no
        + wait
        + numactl --cpunodebind 3 qpid-send -b 127.0.0.1:8888 -a inQ1 -m 1000000 --content-size 300 --capacity 2000 --report-total --sequence no --timestamp no
        tp(m/s)
        37590
        tp(m/s)
        37585
        tp(m/s)
        37453
        tp(m/s)
        37318
        + numactl --cpunodebind 5 qpid-receive -b 127.0.0.1:8888 -a inQ1 -f -m 1000000 --capacity 2000 --ack-frequency 1000 --print-content no --report-total
        + numactl --cpunodebind 4 qpid-receive -b 127.0.0.1:8888 -a inQ1 -f -m 1000000 --capacity 2000 --ack-frequency 1000 --print-content no --report-total
        + numactl --cpunodebind 3 qpid-receive -b 127.0.0.1:8888 -a inQ1 -f -m 1000000 --capacity 2000 --ack-frequency 1000 --print-content no --report-total
        + numactl --cpunodebind 6 qpid-receive -b 127.0.0.1:8888 -a inQ1 -f -m 1000000 --capacity 2000 --ack-frequency 1000 --print-content no --report-total
        + wait
        tp(m/s) l-min l-max l-avg
        41857
        tp(m/s) l-min l-max l-avg
        41839
        tp(m/s) l-min l-max l-avg
        41824
        tp(m/s) l-min l-max l-avg
        41062

        Re-run test, but with receivers and senders running simultaineously. First, pre-patch:

        [kgiusti@xxxx Test1.5]$ ./Test1-Receiver.sh &
        [1] 7886
        [kgiusti@xxxx Test1.5]$ + numactl --cpunodebind 5 qpid-receive -b 127.0.0.1:8888 -a inQ1 -f -m 1000000 --capacity 2000 --ack-frequency 1000 --print-content no --report-total
        + numactl --cpunodebind 4 qpid-receive -b 127.0.0.1:8888 -a inQ1 -f -m 1000000 --capacity 2000 --ack-frequency 1000 --print-content no --report-total
        + numactl --cpunodebind 3 qpid-receive -b 127.0.0.1:8888 -a inQ1 -f -m 1000000 --capacity 2000 --ack-frequency 1000 --print-content no --report-total
        + numactl --cpunodebind 6 qpid-receive -b 127.0.0.1:8888 -a inQ1 -f -m 1000000 --capacity 2000 --ack-frequency 1000 --print-content no --report-total
        + wait
        ./Test1-Sender.sh
        + numactl --cpunodebind 6 qpid-send -b 127.0.0.1:8888 -a inQ1 -m 1000000 --content-size 300 --capacity 2000 --report-total --sequence no --timestamp no
        + numactl --cpunodebind 5 qpid-send -b 127.0.0.1:8888 -a inQ1 -m 1000000 --content-size 300 --capacity 2000 --report-total --sequence no --timestamp no
        + numactl --cpunodebind 4 qpid-send -b 127.0.0.1:8888 -a inQ1 -m 1000000 --content-size 300 --capacity 2000 --report-total --sequence no --timestamp no
        + wait
        + numactl --cpunodebind 3 qpid-send -b 127.0.0.1:8888 -a inQ1 -m 1000000 --content-size 300 --capacity 2000 --report-total --sequence no --timestamp no
        tp(m/s)
        23990
        tp(m/s)
        23920
        tp(m/s)
        23786
        tp(m/s)
        23783
        [kgiusti@xxxx Test1.5]$ tp(m/s) l-min l-max l-avg
        21190
        tp(m/s) l-min l-max l-avg
        21031
        tp(m/s) l-min l-max l-avg
        21008
        tp(m/s) l-min l-max l-avg
        20988

        [1]+ Done ./Test1-Receiver.sh

        Repeat, with patch:

        [kgiusti@xxxx Test1.5]$ + numactl --cpunodebind 5 qpid-receive -b 127.0.0.1:8888 -a inQ1 -f -m 1000000 --capacity 2000 --ack-frequency 1000 --print-content no --report-total
        + numactl --cpunodebind 4 qpid-receive -b 127.0.0.1:8888 -a inQ1 -f -m 1000000 --capacity 2000 --ack-frequency 1000 --print-content no --report-total
        + numactl --cpunodebind 3 qpid-receive -b 127.0.0.1:8888 -a inQ1 -f -m 1000000 --capacity 2000 --ack-frequency 1000 --print-content no --report-total
        + numactl --cpunodebind 6 qpid-receive -b 127.0.0.1:8888 -a inQ1 -f -m 1000000 --capacity 2000 --ack-frequency 1000 --print-content no --report-total
        + wait
        ./Test1-Sender.sh
        + numactl --cpunodebind 6 qpid-send -b 127.0.0.1:8888 -a inQ1 -m 1000000 --content-size 300 --capacity 2000 --report-total --sequence no --timestamp no
        + numactl --cpunodebind 5 qpid-send -b 127.0.0.1:8888 -a inQ1 -m 1000000 --content-size 300 --capacity 2000 --report-total --sequence no --timestamp no
        + numactl --cpunodebind 4 qpid-send -b 127.0.0.1:8888 -a inQ1 -m 1000000 --content-size 300 --capacity 2000 --report-total --sequence no --timestamp no
        + wait
        + numactl --cpunodebind 3 qpid-send -b 127.0.0.1:8888 -a inQ1 -m 1000000 --content-size 300 --capacity 2000 --report-total --sequence no --timestamp no
        tp(m/s)
        25981
        tp(m/s)
        25900
        tp(m/s)
        25814
        tp(m/s)
        25790
        [kgiusti@xxxx Test1.5]$ tp(m/s) l-min l-max l-avg
        22760
        tp(m/s) l-min l-max l-avg
        22733
        tp(m/s) l-min l-max l-avg
        22637
        tp(m/s) l-min l-max l-avg
        22592

        [1]+ Done ./Test1-Receiver.sh

        Show
        Ken Giusti added a comment - Performance of queue fill followed by queue drain, multiple clients, prior to fix: [kgiusti@xxx Test1.5] $ ./Test1-Setup.sh; ./Test1-Sender.sh; ./Test1-Receiver.sh + qpid-config -b 127.0.0.1:8888 add queue inQ1 --max-queue-size=12000000000 --max-queue-count=4000000 --flow-stop-size=0 --flow-stop-count=0 + numactl --cpunodebind 6 qpid-send -b 127.0.0.1:8888 -a inQ1 -m 1000000 --content-size 300 --capacity 2000 --report-total --sequence no --timestamp no + numactl --cpunodebind 5 qpid-send -b 127.0.0.1:8888 -a inQ1 -m 1000000 --content-size 300 --capacity 2000 --report-total --sequence no --timestamp no + numactl --cpunodebind 4 qpid-send -b 127.0.0.1:8888 -a inQ1 -m 1000000 --content-size 300 --capacity 2000 --report-total --sequence no --timestamp no + wait + numactl --cpunodebind 3 qpid-send -b 127.0.0.1:8888 -a inQ1 -m 1000000 --content-size 300 --capacity 2000 --report-total --sequence no --timestamp no tp(m/s) 37135 tp(m/s) 37044 tp(m/s) 37010 tp(m/s) 36945 + numactl --cpunodebind 5 qpid-receive -b 127.0.0.1:8888 -a inQ1 -f -m 1000000 --capacity 2000 --ack-frequency 1000 --print-content no --report-total + numactl --cpunodebind 4 qpid-receive -b 127.0.0.1:8888 -a inQ1 -f -m 1000000 --capacity 2000 --ack-frequency 1000 --print-content no --report-total + numactl --cpunodebind 3 qpid-receive -b 127.0.0.1:8888 -a inQ1 -f -m 1000000 --capacity 2000 --ack-frequency 1000 --print-content no --report-total + numactl --cpunodebind 6 qpid-receive -b 127.0.0.1:8888 -a inQ1 -f -m 1000000 --capacity 2000 --ack-frequency 1000 --print-content no --report-total + wait tp(m/s) l-min l-max l-avg 36776 tp(m/s) l-min l-max l-avg 35813 tp(m/s) l-min l-max l-avg 34623 tp(m/s) l-min l-max l-avg 34519 Same test, with patch to rate limit queue cleanup: [kgiusti@xxxx Test1.5] $ ./Test1-Setup.sh; ./Test1-Sender.sh; ./Test1-Receiver.sh + qpid-config -b 127.0.0.1:8888 add queue inQ1 --max-queue-size=12000000000 --max-queue-count=4000000 --flow-stop-size=0 --flow-stop-count=0 + numactl --cpunodebind 6 qpid-send -b 127.0.0.1:8888 -a inQ1 -m 1000000 --content-size 300 --capacity 2000 --report-total --sequence no --timestamp no + numactl --cpunodebind 5 qpid-send -b 127.0.0.1:8888 -a inQ1 -m 1000000 --content-size 300 --capacity 2000 --report-total --sequence no --timestamp no + numactl --cpunodebind 4 qpid-send -b 127.0.0.1:8888 -a inQ1 -m 1000000 --content-size 300 --capacity 2000 --report-total --sequence no --timestamp no + wait + numactl --cpunodebind 3 qpid-send -b 127.0.0.1:8888 -a inQ1 -m 1000000 --content-size 300 --capacity 2000 --report-total --sequence no --timestamp no tp(m/s) 37590 tp(m/s) 37585 tp(m/s) 37453 tp(m/s) 37318 + numactl --cpunodebind 5 qpid-receive -b 127.0.0.1:8888 -a inQ1 -f -m 1000000 --capacity 2000 --ack-frequency 1000 --print-content no --report-total + numactl --cpunodebind 4 qpid-receive -b 127.0.0.1:8888 -a inQ1 -f -m 1000000 --capacity 2000 --ack-frequency 1000 --print-content no --report-total + numactl --cpunodebind 3 qpid-receive -b 127.0.0.1:8888 -a inQ1 -f -m 1000000 --capacity 2000 --ack-frequency 1000 --print-content no --report-total + numactl --cpunodebind 6 qpid-receive -b 127.0.0.1:8888 -a inQ1 -f -m 1000000 --capacity 2000 --ack-frequency 1000 --print-content no --report-total + wait tp(m/s) l-min l-max l-avg 41857 tp(m/s) l-min l-max l-avg 41839 tp(m/s) l-min l-max l-avg 41824 tp(m/s) l-min l-max l-avg 41062 Re-run test, but with receivers and senders running simultaineously. First, pre-patch: [kgiusti@xxxx Test1.5] $ ./Test1-Receiver.sh & [1] 7886 [kgiusti@xxxx Test1.5] $ + numactl --cpunodebind 5 qpid-receive -b 127.0.0.1:8888 -a inQ1 -f -m 1000000 --capacity 2000 --ack-frequency 1000 --print-content no --report-total + numactl --cpunodebind 4 qpid-receive -b 127.0.0.1:8888 -a inQ1 -f -m 1000000 --capacity 2000 --ack-frequency 1000 --print-content no --report-total + numactl --cpunodebind 3 qpid-receive -b 127.0.0.1:8888 -a inQ1 -f -m 1000000 --capacity 2000 --ack-frequency 1000 --print-content no --report-total + numactl --cpunodebind 6 qpid-receive -b 127.0.0.1:8888 -a inQ1 -f -m 1000000 --capacity 2000 --ack-frequency 1000 --print-content no --report-total + wait ./Test1-Sender.sh + numactl --cpunodebind 6 qpid-send -b 127.0.0.1:8888 -a inQ1 -m 1000000 --content-size 300 --capacity 2000 --report-total --sequence no --timestamp no + numactl --cpunodebind 5 qpid-send -b 127.0.0.1:8888 -a inQ1 -m 1000000 --content-size 300 --capacity 2000 --report-total --sequence no --timestamp no + numactl --cpunodebind 4 qpid-send -b 127.0.0.1:8888 -a inQ1 -m 1000000 --content-size 300 --capacity 2000 --report-total --sequence no --timestamp no + wait + numactl --cpunodebind 3 qpid-send -b 127.0.0.1:8888 -a inQ1 -m 1000000 --content-size 300 --capacity 2000 --report-total --sequence no --timestamp no tp(m/s) 23990 tp(m/s) 23920 tp(m/s) 23786 tp(m/s) 23783 [kgiusti@xxxx Test1.5] $ tp(m/s) l-min l-max l-avg 21190 tp(m/s) l-min l-max l-avg 21031 tp(m/s) l-min l-max l-avg 21008 tp(m/s) l-min l-max l-avg 20988 [1] + Done ./Test1-Receiver.sh Repeat, with patch: [kgiusti@xxxx Test1.5] $ + numactl --cpunodebind 5 qpid-receive -b 127.0.0.1:8888 -a inQ1 -f -m 1000000 --capacity 2000 --ack-frequency 1000 --print-content no --report-total + numactl --cpunodebind 4 qpid-receive -b 127.0.0.1:8888 -a inQ1 -f -m 1000000 --capacity 2000 --ack-frequency 1000 --print-content no --report-total + numactl --cpunodebind 3 qpid-receive -b 127.0.0.1:8888 -a inQ1 -f -m 1000000 --capacity 2000 --ack-frequency 1000 --print-content no --report-total + numactl --cpunodebind 6 qpid-receive -b 127.0.0.1:8888 -a inQ1 -f -m 1000000 --capacity 2000 --ack-frequency 1000 --print-content no --report-total + wait ./Test1-Sender.sh + numactl --cpunodebind 6 qpid-send -b 127.0.0.1:8888 -a inQ1 -m 1000000 --content-size 300 --capacity 2000 --report-total --sequence no --timestamp no + numactl --cpunodebind 5 qpid-send -b 127.0.0.1:8888 -a inQ1 -m 1000000 --content-size 300 --capacity 2000 --report-total --sequence no --timestamp no + numactl --cpunodebind 4 qpid-send -b 127.0.0.1:8888 -a inQ1 -m 1000000 --content-size 300 --capacity 2000 --report-total --sequence no --timestamp no + wait + numactl --cpunodebind 3 qpid-send -b 127.0.0.1:8888 -a inQ1 -m 1000000 --content-size 300 --capacity 2000 --report-total --sequence no --timestamp no tp(m/s) 25981 tp(m/s) 25900 tp(m/s) 25814 tp(m/s) 25790 [kgiusti@xxxx Test1.5] $ tp(m/s) l-min l-max l-avg 22760 tp(m/s) l-min l-max l-avg 22733 tp(m/s) l-min l-max l-avg 22637 tp(m/s) l-min l-max l-avg 22592 [1] + Done ./Test1-Receiver.sh

          People

          • Assignee:
            Ken Giusti
            Reporter:
            Ken Giusti
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development