[ARTEMIS-739] Large messages failing with "(Too many open files)"

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.3.0, 1.4.0
• Fix Version/s: 1.5.0
• Assignee: Justin Bertram
• Reporter: Damien Hollis
• Created: 20/Sep/16 03:38
• Updated: 21/Oct/16 18:42
• Resolved: 28/Sep/16 18:29

Description

We are processing a large number of large messages and many of them are being put into the dead-letter queue as part of a transaction rollback. I suspect that during the rollback the large message is not being released properly and as a result the error below eventually occurs. This may be a more general issue but so far we have only seen this issue when processing messages successfully (although another person in the team mentioned that there seem to be a lot of large messages hanging around).

I noted we are not using the latest version, so I'm in the process of creating a new build and I will test with version 1.4 later today or tomorrow.

org.apache.activemq.artemis.core.server | AMQ222010: Critical IO Error, shutting
down the server. file=NIOSequentialFile /var/data/artemis/large-messages/2147660860.msg, message=/var/data/artemis/large-messages/2147660860.msg (Too many open files)
org.apache.activemq.artemis.api.core.ActiveMQIOErrorException: /var/data/artemis/large-messages/2147660860.msg (Too many open files)
        at org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:101)
        at org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:85)
        at org.apache.activemq.artemis.core.persistence.impl.journal.LargeServerMessageImpl$DecodingContext.open(LargeServerMessageImpl.java:426)
        at org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl$LargeMessageDeliverer.deliver(ServerConsumerImpl.java:1131)
        at org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl.proceedDeliver(ServerConsumerImpl.java:414)
        at org.apache.activemq.artemis.core.server.impl.QueueImpl.proceedDeliver(QueueImpl.java:2464)
        at org.apache.activemq.artemis.core.server.impl.QueueImpl.deliver(QueueImpl.java:1956)
        at org.apache.activemq.artemis.core.server.impl.QueueImpl.access$1500(QueueImpl.java:99)
        at org.apache.activemq.artemis.core.server.impl.QueueImpl$DeliverRunner.run(QueueImpl.java:2695)
        at org.apache.activemq.artemis.utils.OrderedExecutorFactory$OrderedExecutor$ExecutorTask.run(OrderedExecutorFactory.java:103)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.FileNotFoundException: /var/data/artemis/large-messages/2147660860.msg (Too many open files)
        at java.io.RandomAccessFile.open0(Native Method)
        at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243)
        at org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:91)
        ... 12 common frames omitted

Issue Links

• links to: GitHub Pull Request #806