[AMQ-6482] Possible thread leak ?

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 5.14.1
• Fix Version/s: 5.15.0, 5.14.2
• Component/s: Broker
• Labels: threading
• Assignee: Christopher L. Shannon
• Reporter: Maxence Dunnewind
• Created: 26/Oct/16 06:01
• Updated: 04/Nov/16 18:10
• Resolved: 04/Nov/16 17:00

Description

We have an activemq broker running for ~ 24 hours, handling ~ 100k topics and up to 8k connections (for ~ 24k consumers).

Looking at my graph, I see that I get a big increase of threads count happended (in 4 steps, ~ 500 more threads each time) as shown here : https://snapshot.raintank.io/dashboard/snapshot/ExUb4pgNYnmBo92JgbZrvJ6fiBm3PNrI

However, even if the traffic dropped in the night, the thread count never decreased and is still ~ 2500.

I checked a thread dump and it looks like (full dump attached) :

```

1. grep Thread.State thread_dump |awk ' { print $2 }

   ' |sort|uniq -c
   26 RUNNABLE
   70 TIMED_WAITING
   2446 WAITING
   ```

Mhh, so many WAITING threads, let's check why :

```

1. grep 'parking to' thread_dump |cut -d '>' -f 2 |sort |uniq -c
   1 (a java.util.concurrent.CountDownLatch$Sync)
   1308 (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   1159 (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
   41 (a java.util.concurrent.SynchronousQueue$TransferStack)
   ```

I'm not a big java expert but seems a bit unexpected behavior to me.

For various reason (big number of topics etc ) we use custom GC config, here it is :

```
ACTIVEMQ_OPTS_MEMORY="-Xms128M -Xmx10G -Xloggc:/home/log/activemq/gc.log -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseG1GC -XX:InitiatingHeapOccupancyPercent=75 -XX:NewRatio=4"
```

Maxence

Issue Links

• is related to: AMQ-6474 Memory leak