Details
Major Description
There's about 20 topics with virtual topic enabled, hundreds of comsumers/producers connected to MQ on NIO transport connector. During the run there're about 12000 msg flow in per second, not a very high rate, but ActiveMQ consumes a lot of CPU resource (about 600%~1000%). To find out what's the most CPU consuming code path, I use JProfiler to dig into the process.
Among all the NIO worker threads, most of them were frequently blocked and did a little job between the 'unblocked' time. While they're expected spend most of their time slices on waiting for work item and processed them.
After reviewing the monitor usage history and stats, I think these NIO workers were competing fiercely with each other on executing a synchronized method (DestinationMap::get), which is also the most hot spot in the program . I also notice that the caller AbstractRegion::getDestinations acquires a read lock before calling it, so I guess this could be a left out, read lock is the actual lock type required here.
It's too difficult for me to list all critical sections between NIO workers, or between NIO workers and BrokerService which adds up to the overall synchronize overhead. So I attach the relevant info, with the hope of finding a complete solution to this.
