Major Description
The following exception is thrown from an ActiveMQ instance that is in Slave mode that causes it to crash.
java.io.FileNotFoundException: **/*****/***/******/journal/control.dat (Too many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
at org.apache.activeio.journal.active.ControlFile.<init>(ControlFile.java:60)
at org.apache.activeio.journal.active.LogFileManager.initialize(LogFileManager.java:119)
at org.apache.activeio.journal.active.LogFileManager.<init>(LogFileManager.java:102)
at org.apache.activeio.journal.active.JournalImpl.<init>(JournalImpl.java:101)
at org.apache.activemq.store.DefaultPersistenceAdapterFactory.createJournal(DefaultPersistenceAdapterFactory.java:224)
at org.apache.activemq.store.DefaultPersistenceAdapterFactory.getJournal(DefaultPersistenceAdapterFactory.java:132)
at org.apache.activemq.store.DefaultPersistenceAdapterFactory.createPersistenceAdapter(DefaultPersistenceAdapterFactory.java:70)
at org.apache.activemq.store.PersistenceAdapterFactoryBean.getObject(PersistenceAdapterFactoryBean.java:35)
at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectFromFactoryBean(AbstractBeanFactory.java:994)
... 41 more
The problem is that each time ActiveIO checks to see if control.dat is locked it creates a new java.io.RandomAccessFile. If the control.dat file is locked, it fails to close the RandomAccessFile.
This exception always will be thrown from an instance in Shared-File-System Slave mode that continues to check the Journal file (which the Master has locked). But the rate at which is is thrown depends on the maximum number of files the OS allows to be open. You can check how many files are open with 'ulimit -a'