[QPID-5334] Low throughput with thousands of queues

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 0.24
• Component/s: C++ Broker, JMS AMQP 0-x
• Labels: performance

Environment

Broker: Linux 2.6.32 64bit
Client: Windows 7, Java 1.7.0_21 64bit

• Assignee: Unassigned
• Reporter: Flavio Pellanda
• Created: 12/Nov/13 16:37
• Updated: 14/Mar/18 02:39
• Resolved: 30/Jan/18 17:22

Description

I have an use case where I need to create hundreds of thousands of queues,
each one subscribed to a different topic (therefore I have as many topics
as queues).
I set up a test with a single producer generating data on a randomly
chosen topic, and a receiver retrieving data from the queues (and throwing
it away).

I'm using the JMS api, and doing the obvious thing makes the throughput
drop dramatically from 10k msg/sec with a single topic/queue (around the
top my network adapter can sustain) to 20 msg/sec with 100k topics/queues.
I found out that I can recover performance by using more JMS sessions and
connections - e.g. create 4 connections with 100 sessions each, and
randomly distributing the receiving queues on them.
This however is less than ideal, since with the JMS client a thread is
created for each session, I can manage the workload with 1 thread only when it's over a single queue.