Description
Non-persistent messages either from high message producer or slow consumer left unchecked could cause a file handle leak. The effect is many files being created under directory /<activemq_install>/data/leveldb/plist.index and ActiveMQ become unresponsive to where it cannot processing any messages until its process is bounced.
2015-06-01 10:58:21,674 | ERROR | Caught an Exception adding a message: ActiveMQTextMessage {commandId = 6904, responseRequired = false, messageId = ID:testico180l-45369-1432936489601-1:83688:1:1:6900, originalDestination = null, originalTransactionId = null, producerId = ID:testico180l-45369-1432936489601-1:83688:1:1, destination = queue://FOX.DEV.ERROR, transactionId = null, expiration = 0, timestamp = 1433174297512, arrival = 0, brokerInTime = 1433174301674, brokerOutTime = 0, correlationId = null, replyTo = null, persistent = false, type = null, priority = 4, groupID = null, groupSequence = 0, targetConsumerId = null, compressed = false, userID = null, content = org.apache.activemq.util.ByteSequence@10a9ac16, marshalledProperties = org.apache.activemq.util.ByteSequence@7360e0e5, dataStructure = null, redeliveryCounter = 0, size = 0, properties =
{umh_call_chain=testico189l\testico189l_CIT\AMQ\20150601105821674\queue://FOX.DEV.ERROR, umh_origin_host=testico180l, umh_process_key=activemq_client, umh_application_version=5.11.1, umh_application_id=ActiveMQ, umh_routing_environment=DEV, umh_header_version=1.0.0, umh_routing_sub_environment=DEV, umh_message_id=EF81B0F3BD3740E1941EC1C44AB2F6D5, umh_release_id=20150522000000, tgenv=DEV, umh_channel=MQ}, readOnlyProperties = false, readOnlyBody = false, droppable = false, jmsXGroupFirstForConsumer = false, text = Test message} first to FilePendingMessageCursor | org.apache.activemq.broker.region.cursors.FilePendingMessageCursor | ActiveMQ NIO Worker 2224
org.iq80.leveldb.impl.DbImpl$BackgroundProcessingException: java.io.FileNotFoundException: /activemq/data/leveldb/plist.index/291781.sst (Too many open files)
at org.iq80.leveldb.impl.DbImpl.checkBackgroundException(DbImpl.java:411)[leveldb-0.6.jar:]
at org.iq80.leveldb.impl.DbImpl.writeInternal(DbImpl.java:664)[leveldb-0.6.jar:]
at org.iq80.leveldb.impl.DbImpl.put(DbImpl.java:630)[leveldb-0.6.jar:]
at org.apache.activemq.leveldb.LevelDBClient$RichDB.put(LevelDBClient.scala:224)[activemq-leveldb-store-5.11.1.jar:5.11.1]
at org.apache.activemq.leveldb.LevelDBClient.plistPut(LevelDBClient.scala:1716)[activemq-leveldb-store-5.11.1.jar:5.11.1]
at org.apache.activemq.leveldb.DBManager.plistPut(DBManager.scala:884)[activemq-leveldb-store-5.11.1.jar:5.11.1]
at org.apache.activemq.leveldb.LevelDBStore$LevelDBPList.add(LevelDBStore.scala:1050)[activemq-leveldb-store-5.11.1.jar:5.11.1]
at org.apache.activemq.leveldb.LevelDBStore$LevelDBPList.addLast(LevelDBStore.scala:1037)[activemq-leveldb-store-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.tryAddMessageLast(FilePendingMessageCursor.java:237)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.addMessageLast(FilePendingMessageCursor.java:207)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.region.cursors.StoreQueueCursor.addMessageLast(StoreQueueCursor.java:97)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.region.Queue.cursorAdd(Queue.java:1796)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.region.Queue.orderedCursorAdd(Queue.java:878)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:854)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.region.Queue.send(Queue.java:733)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:419)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:468)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.jmx.ManagedRegionBroker.send(ManagedRegionBroker.java:297)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:152)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.scheduler.SchedulerBroker.send(SchedulerBroker.java:169)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:152)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:96)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:307)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:152)[activemq-broker-5.11.1.jar:5.11.1]
at main.com.test.activemq.broker.plugin.umh.UnifiedMessageHeaderFilter.send(UnifiedMessageHeaderFilter.java:417)[test_activemq_broker_plugins.jar:]
at org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:157)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:541)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:768)[activemq-client-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:334)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:188)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)[activemq-client-5.11.1.jar:5.11.1]
at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)[activemq-client-5.11.1.jar:5.11.1]
at org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:270)[activemq-client-5.11.1.jar:5.11.1]
at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)[activemq-client-5.11.1.jar:5.11.1]
at org.apache.activemq.transport.nio.NIOTransport.serviceRead(NIOTransport.java:138)[activemq-client-5.11.1.jar:5.11.1]
at org.apache.activemq.transport.nio.NIOTransport$1.onSelect(NIOTransport.java:69)[activemq-client-5.11.1.jar:5.11.1]
at org.apache.activemq.transport.nio.SelectorSelection.onSelect(SelectorSelection.java:97)[activemq-client-5.11.1.jar:5.11.1]
at org.apache.activemq.transport.nio.SelectorWorker$1.run(SelectorWorker.java:119)[activemq-client-5.11.1.jar:5.11.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)[:1.7.0_75]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)[:1.7.0_75]
at java.lang.Thread.run(Thread.java:745)[:1.7.0_75]
Attachments
Issue Links
- is duplicated by
-
AMQ-5404 LevelDB files not being released after deletion
-
- Resolved
-