UIMA
  1. UIMA
  2. UIMA-1766

Fix Memory Leak in UIMA AS Service

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.3AS
    • Fix Version/s: 2.3.1AS
    • Component/s: Async Scaleout
    • Labels:
      None

      Description

      When running an Aggregate with a colocated Cas Multiplier per CAS statistics are not being removed from a Map maintained by the Cas Multiplier service. As more CASes are sent to the CM, instances of ServicePerformance object are added to the Map and never removed leading to a memory leak and eventual OOM error.

        Activity

        Hide
        Jerry Cwiklik added a comment -

        Modified dropCasStatistics() in BaseAnalysisEngineController to stop checking if a service is top level. The method always removes an entry from the perCasStatistics Map. Jorn, check if this fix works for you

        Show
        Jerry Cwiklik added a comment - Modified dropCasStatistics() in BaseAnalysisEngineController to stop checking if a service is top level. The method always removes an entry from the perCasStatistics Map. Jorn, check if this fix works for you
        Hide
        Joern Kottmann added a comment -

        My test shows that the issue is fixed now. A test instance analyzed 650K CASes and in the memory dump only 38 instances of the ServicePerformance object exist.

        Show
        Joern Kottmann added a comment - My test shows that the issue is fixed now. A test instance analyzed 650K CASes and in the memory dump only 38 instances of the ServicePerformance object exist.
        Hide
        Jerry Cwiklik added a comment -

        Jorn found another memory leak in the UIMA AS service. The new one is caused by UIMA AS service not removing entries from a Map maintaining statistics for colocated delegates.

        Show
        Jerry Cwiklik added a comment - Jorn found another memory leak in the UIMA AS service. The new one is caused by UIMA AS service not removing entries from a Map maintaining statistics for colocated delegates.
        Hide
        Jerry Cwiklik added a comment -

        Modified UIMA AS code to remove entry in a Map maintaining statistics (statsMap) after a CAS is dispatched to a colocated component. Jorn, let me know if this fix works.

        Show
        Jerry Cwiklik added a comment - Modified UIMA AS code to remove entry in a Map maintaining statistics (statsMap) after a CAS is dispatched to a colocated component. Jorn, let me know if this fix works.
        Hide
        Joern Kottmann added a comment - - edited

        I think the first fix broke something, when the current trunk is run on our server it spams the log file with
        this exception traces. The test server the service is running on has the ip 10.45.2.103 and our test activemq broker
        is running on 10.45.2.100.

        It looks like it tries to use the activemq broker for some JMX stuff.

        Here is the error message:
        4/20/10 4:20:56 PM - 30: org.apache.uima.adapter.jms.activemq.JmsInputChannel.ackMessage: WARNING:
        java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 10.45.2.100; nested exception is:
        java.net.ConnectException: Connection refused]
        at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:323)
        at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:248)
        at org.apache.uima.aae.jmx.RemoteJMXServer.initialize(RemoteJMXServer.java:76)
        at org.apache.uima.adapter.jms.activemq.JmsInputChannel.attachToRemoteBrokerJMXServer(JmsInputChannel.java:837)
        at org.apache.uima.adapter.jms.activemq.JmsInputChannel.onMessage(JmsInputChannel.java:605)
        at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:518)
        at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:479)
        at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:451)
        at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:323)
        at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:261)
        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:982)
        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:974)
        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:876)
        at java.lang.Thread.run(Thread.java:619)
        Caused by: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 10.45.2.100; nested exception is:
        java.net.ConnectException: Connection refused]
        at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:101)
        at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:185)
        at javax.naming.InitialContext.lookup(InitialContext.java:392)
        at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1871)
        at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1841)
        at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:257)
        ... 13 more
        Caused by: java.rmi.ConnectException: Connection refused to host: 10.45.2.100; nested exception is:
        java.net.ConnectException: Connection refused
        at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:601)
        at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:198)
        at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184)
        at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:322)
        at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
        at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:97)
        ... 18 more
        Caused by: java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
        at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
        at java.net.Socket.connect(Socket.java:519)
        at java.net.Socket.connect(Socket.java:469)
        at java.net.Socket.<init>(Socket.java:366)
        at java.net.Socket.<init>(Socket.java:180)
        at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:22)
        at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:128)
        at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:595)
        ... 23 more

        Show
        Joern Kottmann added a comment - - edited I think the first fix broke something, when the current trunk is run on our server it spams the log file with this exception traces. The test server the service is running on has the ip 10.45.2.103 and our test activemq broker is running on 10.45.2.100. It looks like it tries to use the activemq broker for some JMX stuff. Here is the error message: 4/20/10 4:20:56 PM - 30: org.apache.uima.adapter.jms.activemq.JmsInputChannel.ackMessage: WARNING: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 10.45.2.100; nested exception is: java.net.ConnectException: Connection refused] at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:323) at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:248) at org.apache.uima.aae.jmx.RemoteJMXServer.initialize(RemoteJMXServer.java:76) at org.apache.uima.adapter.jms.activemq.JmsInputChannel.attachToRemoteBrokerJMXServer(JmsInputChannel.java:837) at org.apache.uima.adapter.jms.activemq.JmsInputChannel.onMessage(JmsInputChannel.java:605) at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:518) at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:479) at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:451) at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:323) at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:261) at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:982) at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:974) at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:876) at java.lang.Thread.run(Thread.java:619) Caused by: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 10.45.2.100; nested exception is: java.net.ConnectException: Connection refused] at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:101) at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:185) at javax.naming.InitialContext.lookup(InitialContext.java:392) at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1871) at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1841) at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:257) ... 13 more Caused by: java.rmi.ConnectException: Connection refused to host: 10.45.2.100; nested exception is: java.net.ConnectException: Connection refused at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:601) at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:198) at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184) at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:322) at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source) at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:97) ... 18 more Caused by: java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:519) at java.net.Socket.connect(Socket.java:469) at java.net.Socket.<init>(Socket.java:366) at java.net.Socket.<init>(Socket.java:180) at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:22) at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:128) at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:595) ... 23 more
        Hide
        Joern Kottmann added a comment -

        The problem described in the last comment is actually linked to UIMA-1726 and not to this issue.

        Show
        Joern Kottmann added a comment - The problem described in the last comment is actually linked to UIMA-1726 and not to this issue.
        Hide
        Joern Kottmann added a comment -

        It seems that now all memory leaks are fixed, the memory consumption did not increase after processing 350K CASes in our test system with the latest revision.

        Show
        Joern Kottmann added a comment - It seems that now all memory leaks are fixed, the memory consumption did not increase after processing 350K CASes in our test system with the latest revision.

          People

          • Assignee:
            Jerry Cwiklik
            Reporter:
            Jerry Cwiklik
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development