[AMQ-4872] ActiveMQ client loses connectivity and admin interface is corrupted after being up for ~60 hours

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Invalid
• Affects Version/s: 5.5.0
• Fix Version/s: 5.10.0

Environment

Centos 6.4. ActiveMQ running at nice 0, several clients running at nice 10 and one running at nice -10.

• Assignee: Claus Ibsen
• Reporter: John Griffith
• Created: 11/Nov/13 21:32
• Updated: 12/Nov/13 10:41
• Resolved: 12/Nov/13 10:41

Description

We run an application that, at the moment, creates and uses three queues accessed via Spring JMSTemplate: Ingest, IngestRequest, and Index.

ActiveMQ ran fine for a period of about 60 hours. The system came under heavy, continuous load at about the 55 hour mark. At about the 60 hour point, the high-priority client (using spring jmsTemplate) lost it's connection to the ActiveMQ server, reporting a jms ConnectionFailedException. On the server side, ActiveMQ logged the following for each attempt to connect by the client:

2013-11-11 11:50:12,667 [127.0.0.1:48682] INFO Transport - Transport failed: java.io.EOFException

In addition, accessing the queues in the web administrative interface shows a list of 'garbage' entries that includes the list of files that appear to be from a cgi directory, but includes .asp files that one wouldn't find on a Linux host.

Running 'service activemq status' reports activemq as being up. Clients can not connect to it, however. There is plenty of disk space and memory available, however there are long-running (minutes) tasks that could conceivably starve ActiveMQ for CPU time - these tasks all run at the higher nice value.