ActiveMQ
  1. ActiveMQ
  2. AMQ-1380

ActiveIO does not close ControlFile

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 4.1.1
    • Fix Version/s: 5.3.0
    • Component/s: Broker
    • Labels:
      None
    • Environment:

      All

    • Patch Info:
      Patch Available

      Description

      The following exception is thrown from an ActiveMQ instance that is in Slave mode that causes it to crash.

      java.io.FileNotFoundException: **/*****/***/******/journal/control.dat (Too many open files)
      at java.io.RandomAccessFile.open(Native Method)
      at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
      at org.apache.activeio.journal.active.ControlFile.<init>(ControlFile.java:60)
      at org.apache.activeio.journal.active.LogFileManager.initialize(LogFileManager.java:119)
      at org.apache.activeio.journal.active.LogFileManager.<init>(LogFileManager.java:102)
      at org.apache.activeio.journal.active.JournalImpl.<init>(JournalImpl.java:101)
      at org.apache.activemq.store.DefaultPersistenceAdapterFactory.createJournal(DefaultPersistenceAdapterFactory.java:224)
      at org.apache.activemq.store.DefaultPersistenceAdapterFactory.getJournal(DefaultPersistenceAdapterFactory.java:132)
      at org.apache.activemq.store.DefaultPersistenceAdapterFactory.createPersistenceAdapter(DefaultPersistenceAdapterFactory.java:70)
      at org.apache.activemq.store.PersistenceAdapterFactoryBean.getObject(PersistenceAdapterFactoryBean.java:35)
      at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectFromFactoryBean(AbstractBeanFactory.java:994)
      ... 41 more

      The problem is that each time ActiveIO checks to see if control.dat is locked it creates a new java.io.RandomAccessFile. If the control.dat file is locked, it fails to close the RandomAccessFile.

      This exception always will be thrown from an instance in Shared-File-System Slave mode that continues to check the Journal file (which the Master has locked). But the rate at which is is thrown depends on the maximum number of files the OS allows to be open. You can check how many files are open with 'ulimit -a'

        Activity

        Jeff Turner made changes -
        Project Import Fri Nov 26 22:32:02 EST 2010 [ 1290828722158 ]
        Dejan Bosanac made changes -
        Resolution Fixed [ 1 ]
        Status Reopened [ 4 ] Resolved [ 5 ]
        Hide
        Dejan Bosanac added a comment -

        ActiveIO 3.1.2 has just been released with this patch included. It will be included by default in 5.3.0. For earlier versions, download the jar from https://repository.apache.org/content/repositories/releases/org/apache/activemq/activeio-core/3.1.2/ and put it in lib/optional folder.

        Show
        Dejan Bosanac added a comment - ActiveIO 3.1.2 has just been released with this patch included. It will be included by default in 5.3.0. For earlier versions, download the jar from https://repository.apache.org/content/repositories/releases/org/apache/activemq/activeio-core/3.1.2/ and put it in lib/optional folder.
        Dejan Bosanac made changes -
        Assignee Dejan Bosanac [ dejanb ]
        Rob Davies made changes -
        Fix Version/s 5.3.0 [ 11914 ]
        Fix Version/s 5.1.0 [ 11802 ]
        Hide
        Douglas Miranda added a comment - - edited

        hello Friends!

        I'd like to know how can I apply that patch? I've tried to find a way, but I can't find out anything.

        can anybody give some help?

        Thank you!

        Show
        Douglas Miranda added a comment - - edited hello Friends! I'd like to know how can I apply that patch? I've tried to find a way, but I can't find out anything. can anybody give some help? Thank you!
        Vadim Katz made changes -
        Resolution Fixed [ 1 ]
        Status Resolved [ 5 ] Reopened [ 4 ]
        Hide
        Vadim Katz added a comment -

        It does not look like the patch made it into activeio-core-3.1.0 which is included in 5.1.0

        Show
        Vadim Katz added a comment - It does not look like the patch made it into activeio-core-3.1.0 which is included in 5.1.0
        Hiram Chirino made changes -
        Fix Version/s 5.2.0 [ 11841 ]
        Fix Version/s 5.1.0 [ 11802 ]
        Hiram Chirino made changes -
        Field Original Value New Value
        Fix Version/s 5.2.0 [ 11841 ]
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        Hiram Chirino added a comment -

        Patch applied in rev 646352

        Show
        Hiram Chirino added a comment - Patch applied in rev 646352
        Hide
        Shahram Javey added a comment -

        Why the activeio-core in ActiveMQ lib (activeio-core-3.1-SNAPSHOT.jar) is still labeled as SNAPSHOT? is this a bug in the build label? It is odd to have 5.0 as a released version but it still includes a SNAPSHOT jar.

        Show
        Shahram Javey added a comment - Why the activeio-core in ActiveMQ lib (activeio-core-3.1-SNAPSHOT.jar) is still labeled as SNAPSHOT? is this a bug in the build label? It is odd to have 5.0 as a released version but it still includes a SNAPSHOT jar.
        Hide
        Shahram Javey added a comment - - edited

        This patch is still not part of ActiveMQ 5.0, where slave file handle leaks are still an issue. I manually applied this patch and it fixed the problem. Why is this patch not part of 5.0? Running a master/slave (persistence based on journal file-system) using a conf file below, leads to the crash of the slave after a few hours, due to running out of open files. If you monitor the number of open files (lsof | wc -l), you'll see that the number of open files keeps increasing every 10 seconds. With this patch, this is no-longer a problem. Please include this patch in the release. Here is the conf file that I've used for master/slave (the only difference is that the value of the IP address in the broker name and the transport connector)

        <beans
        xmlns="http://www.springframework.org/schema/beans"
        xmlns:amq="http://activemq.org/config/1.0"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
        http://activemq.org/config/1.0 http://activemq.apache.org/schema/activemq-core.xsd
        http://activemq.apache.org/camel/schema/spring http://activemq.apache.org/camel/schema/spring/camel-spring.xsd">

        <!-- Allows us to use system properties as variables in this configuration file -->
        <bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer"/>

        <broker xmlns="http://activemq.org/config/1.0" brokerName="10.0.1.200" useJmx="false">

        <!-- The transport connectors ActiveMQ will listen to -->
        <transportConnectors>
        <transportConnector name="openwire" uri="tcp://10.0.1.200:61616?trace=true"/>
        </transportConnectors>
        <persistenceAdapter>
        <journaledJDBC dataDirectory="/Users/sjavey/amq-broker-data"/>
        </persistenceAdapter>

        </broker>

        <!-- An embedded servlet engine for serving up the Admin console -->
        <jetty xmlns="http://mortbay.com/schemas/jetty/1.0">
        <connectors>
        <nioConnector port="8161" />
        </connectors>

        <handlers>
        <webAppContext contextPath="/admin" resourceBase="$

        {activemq.base}

        /webapps/admin" logUrlOnStart="true" />
        </handlers>
        </jetty>

        </beans>

        Show
        Shahram Javey added a comment - - edited This patch is still not part of ActiveMQ 5.0, where slave file handle leaks are still an issue. I manually applied this patch and it fixed the problem. Why is this patch not part of 5.0? Running a master/slave (persistence based on journal file-system) using a conf file below, leads to the crash of the slave after a few hours, due to running out of open files. If you monitor the number of open files (lsof | wc -l), you'll see that the number of open files keeps increasing every 10 seconds. With this patch, this is no-longer a problem. Please include this patch in the release. Here is the conf file that I've used for master/slave (the only difference is that the value of the IP address in the broker name and the transport connector) <beans xmlns="http://www.springframework.org/schema/beans" xmlns:amq="http://activemq.org/config/1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd http://activemq.org/config/1.0 http://activemq.apache.org/schema/activemq-core.xsd http://activemq.apache.org/camel/schema/spring http://activemq.apache.org/camel/schema/spring/camel-spring.xsd "> <!-- Allows us to use system properties as variables in this configuration file --> <bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer"/> <broker xmlns="http://activemq.org/config/1.0" brokerName="10.0.1.200" useJmx="false"> <!-- The transport connectors ActiveMQ will listen to --> <transportConnectors> <transportConnector name="openwire" uri="tcp://10.0.1.200:61616?trace=true"/> </transportConnectors> <persistenceAdapter> <journaledJDBC dataDirectory="/Users/sjavey/amq-broker-data"/> </persistenceAdapter> </broker> <!-- An embedded servlet engine for serving up the Admin console --> <jetty xmlns="http://mortbay.com/schemas/jetty/1.0"> <connectors> <nioConnector port="8161" /> </connectors> <handlers> <webAppContext contextPath="/admin" resourceBase="$ {activemq.base} /webapps/admin" logUrlOnStart="true" /> </handlers> </jetty> </beans>
        Hide
        Lilians AUVIGNE added a comment -

        I'm having the same problem.

        This patch solve it.

        Could you add this issue at 4.1.2 RoadMap.

        Thanks Joe for this patch.

        Show
        Lilians AUVIGNE added a comment - I'm having the same problem. This patch solve it. Could you add this issue at 4.1.2 RoadMap. Thanks Joe for this patch.
        Joe Kutner created issue -

          People

          • Assignee:
            Dejan Bosanac
            Reporter:
            Joe Kutner
          • Votes:
            7 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development