Uploaded image for project: 'James Server'
  1. James Server
  2. JAMES-512

OutOfMemoryError -- object not released.

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.2.0
    • Fix Version/s: 2.3.0
    • Labels:
      None
    • Environment:
      windows XP,
      Java 1.5.0_06

      Description

      I use the file system to store the emails.

      When I run the James for a long time, the james server used more and more heap memory, eventually it runs out of memory and refuse to receive email. only restart the james will work.

      I used JProfile Memory Debugger(you can get the trial version from www.quest.com) and found a clue. the object allocated from following code at line 92 does not always get released, cause the memory been used up eventually. I will attach a picture file showing the call stack.

      91, final OutputStream outputStream = getOutputStream( key );
      92, final BufferedOutputStream stream = new BufferedOutputStream( outputStream );
      93
      94, final Object o = m_outputs.get( key );
      95, if( null == o )
      96,

      { 97, m_outputs.put( key; stream ); 98, }

      99, else if( o instanceof ArrayList )
      100,

      { 101, ( (ArrayList)o ).add( stream ); 102, }

      103, else
      104,

      { 105, final ArrayList list = new ArrayList(); 106, list.add( o ); 107, list.add( stream ); 108, m_outputs.put( key; list ); 109, }

      110
      111, return stream;

      1. james-512.patch
        11 kB
        Stefano Bagnara
      2. picture.GIF
        18 kB
        Quande Ren

        Activity

        Hide
        r_q_d Quande Ren added a comment -

        Call stack for the memory leak

        Show
        r_q_d Quande Ren added a comment - Call stack for the memory leak
        Hide
        r_q_d Quande Ren added a comment -

        The code is from org\apache\james\mailrepository\filepair\File_Persistent_Object_Repository.java

        Show
        r_q_d Quande Ren added a comment - The code is from org\apache\james\mailrepository\filepair\File_Persistent_Object_Repository.java
        Hide
        bago Stefano Bagnara added a comment -

        Can you search the logs for these strings:
        "Exception caught while storing a stream" WARN level (maybe in objectstorage)
        "Exception storing mail" ERROR level in (maybe in mailstore)

        From a code review I think the only leak could be when an exception is thrown in that code and "stream" has been created but not returned.

        We should probably add only a "if (stream != null) stream.close()" in the catch clause at the end of the code you reported.

        Can you provide this log info and eventually try to patch the code with the above line and test again?

        Show
        bago Stefano Bagnara added a comment - Can you search the logs for these strings: "Exception caught while storing a stream" WARN level (maybe in objectstorage) "Exception storing mail" ERROR level in (maybe in mailstore) From a code review I think the only leak could be when an exception is thrown in that code and "stream" has been created but not returned. We should probably add only a "if (stream != null) stream.close()" in the catch clause at the end of the code you reported. Can you provide this log info and eventually try to patch the code with the above line and test again?
        Hide
        brainlounge Bernd Fondermann added a comment -

        I'd like to have some more info to eventually reproduce the problem:

        o How long does James run before it runs out of heap?
        o What is your max heap?
        o I assume from your bug report that you are using James 2.2.0?
        o How much email load do you get, approximately?
        o How many user accounts do you have, approximately?

        Thank you!

        Show
        brainlounge Bernd Fondermann added a comment - I'd like to have some more info to eventually reproduce the problem: o How long does James run before it runs out of heap? o What is your max heap? o I assume from your bug report that you are using James 2.2.0? o How much email load do you get, approximately? o How many user accounts do you have, approximately? Thank you!
        Hide
        r_q_d Quande Ren added a comment -

        the code "if (stream != null) stream.close()" is already presented at line 287 in the AvalonMailRepository.java. So I don't think this is the reason for OutOfMemoryError.

        For the code final BufferedOutputStream stream = new BufferedOutputStream( outputStream ); , it will allocate a buffer(byte[8192]), and this stream will be stored in a hashtable (line 97, or 101, or 108.). but when did this stream get removed from the hashtable? If it nener get removed from the hashtable, then this is the place where it causes the OutOfMemoryError.

        Show
        r_q_d Quande Ren added a comment - the code "if (stream != null) stream.close()" is already presented at line 287 in the AvalonMailRepository.java. So I don't think this is the reason for OutOfMemoryError. For the code final BufferedOutputStream stream = new BufferedOutputStream( outputStream ); , it will allocate a buffer(byte [8192] ), and this stream will be stored in a hashtable (line 97, or 101, or 108.). but when did this stream get removed from the hashtable? If it nener get removed from the hashtable, then this is the place where it causes the OutOfMemoryError.
        Hide
        r_q_d Quande Ren added a comment -

        I used James 2.2.0.

        max heap is 128m.

        only have 1 user account. ( I tested it to send email from this account to itself.)

        It begin to get OutOfMemoryError shows up in the spoolmanager-yyyy-mm-dd-hh-MM.log after sending about 7630 messages in 15 minutes. ( the message is about 10 bytes for subject and 10 bytes for content.).

        Show
        r_q_d Quande Ren added a comment - I used James 2.2.0. max heap is 128m. only have 1 user account. ( I tested it to send email from this account to itself.) It begin to get OutOfMemoryError shows up in the spoolmanager-yyyy-mm-dd-hh-MM.log after sending about 7630 messages in 15 minutes. ( the message is about 10 bytes for subject and 10 bytes for content.).
        Hide
        bago Stefano Bagnara added a comment -

        Quande,
        the stream.close in the AvalonMailRepository is there but is not enough when the exception is thrown before the stream is returned by the File_Persistent_Object_Repository. So we should add that check even in its catch.

        Btw after your suggestion and further checks I think you are right and the problem is the buffers of the "cached" BufferedOutputStream.

        We should add WeakReferences for that cache.

        I don't even know why that hashmap is there, so before making any change I should try to understand why.

        I'll do that tomorrow...

        Show
        bago Stefano Bagnara added a comment - Quande, the stream.close in the AvalonMailRepository is there but is not enough when the exception is thrown before the stream is returned by the File_Persistent_Object_Repository. So we should add that check even in its catch. Btw after your suggestion and further checks I think you are right and the problem is the buffers of the "cached" BufferedOutputStream. We should add WeakReferences for that cache. I don't even know why that hashmap is there, so before making any change I should try to understand why. I'll do that tomorrow...
        Hide
        r_q_d Quande Ren added a comment -

        I checked after sending 200 emails, when the email moved from the spool to outgoing to the inboxes, around 200*2*8192 bytes get remain in memory and never get gc'ed.

        I think there should be a place to remove both the key and stream, not just using WeakReferences to remove the stream.

        Show
        r_q_d Quande Ren added a comment - I checked after sending 200 emails, when the email moved from the spool to outgoing to the inboxes, around 200*2*8192 bytes get remain in memory and never get gc'ed. I think there should be a place to remove both the key and stream, not just using WeakReferences to remove the stream.
        Hide
        bago Stefano Bagnara added a comment -

        2 possible solutions:

        1. Remove the whole m_inputs, m_outputs HashMap. If we correctly close the streams all around James it should work fine. I can only see a problem with remove being called before message disposition. We can only know if we test it.

        2. Change the current HashMap "message key" => "stream" to use weak references and to track when streams are closed. This is much more difficult.

        I understand you know where to look (you found the leak), so I hope you can give a try to the "1." solution.

        Show
        bago Stefano Bagnara added a comment - 2 possible solutions: 1. Remove the whole m_inputs, m_outputs HashMap. If we correctly close the streams all around James it should work fine. I can only see a problem with remove being called before message disposition. We can only know if we test it. 2. Change the current HashMap "message key" => "stream" to use weak references and to track when streams are closed. This is much more difficult. I understand you know where to look (you found the leak), so I hope you can give a try to the "1." solution.
        Hide
        r_q_d Quande Ren added a comment -

        Hi, Stefano,

        Here is the changed version of File_Persistent_Object_Repository.java according to your suggestion No. 1. I tested the basic functions according to my knowledge and it is working. There's no memory leak anymore.

        Let me know your comments.

        Quande

        /***********************************************************************

        • Copyright (c) 2000-2004 The Apache Software Foundation. *
        • All rights reserved. *
        • ------------------------------------------------------------------- *
        • Licensed under the Apache License, Version 2.0 (the "License"); you *
        • may not use this file except in compliance with the License. You *
        • may obtain a copy of the License at: *
        • *
        • http://www.apache.org/licenses/LICENSE-2.0 *
        • *
        • Unless required by applicable law or agreed to in writing, software *
        • distributed under the License is distributed on an "AS IS" BASIS, *
        • WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or *
        • implied. See the License for the specific language governing *
        • permissions and limitations under the License. *
          ***********************************************************************/

        package org.apache.james.mailrepository.filepair;

        import java.io.BufferedOutputStream;
        import java.io.IOException;
        import java.io.InputStream;
        import java.io.OutputStream;
        import java.util.ArrayList;
        import java.util.HashMap;
        import org.apache.avalon.cornerstone.services.store.StreamRepository;
        import org.apache.avalon.excalibur.io.IOUtil;

        /**

        • Implementation of a StreamRepository to a File.
        • TODO: -retieve(String key) should return a FilterInputStream to allow
        • mark and reset methods. (working not like BufferedInputStream!!!)
          *
          */
          public class File_Persistent_Stream_Repository
          extends AbstractFileRepository
          implements StreamRepository
          {
          //protected final HashMap m_inputs = new HashMap();
          //protected final HashMap m_outputs = new HashMap();

        protected String getExtensionDecorator()

        { return ".FileStreamStore"; }

        /**

        • Get the object associated to the given unique key.
          */
          public synchronized InputStream get( final String key )
          {
          try
          {
          final ResettableFileInputStream stream =
          new ResettableFileInputStream( getFile( key ) );

        /*
        final Object o = m_inputs.get( key );
        if( null == o )

        { m_inputs.put( key, stream ); }

        else if( o instanceof ArrayList )

        { ( (ArrayList)o ).add( stream ); }
        else
        { final ArrayList list = new ArrayList(); list.add( o ); list.add( stream ); m_inputs.put( key, list ); }
        */
        return stream;
        }
        catch( final IOException ioe )
        { final String message = "Exception caught while retrieving a stream "; getLogger().warn( message, ioe ); throw new RuntimeException( message + ": " + ioe ); }
        }

        /**
        * Store the given object and associates it to the given key
        */
        public synchronized OutputStream put( final String key )
        {
        try
        {
        final OutputStream outputStream = getOutputStream( key );
        final BufferedOutputStream stream = new BufferedOutputStream( outputStream );
        /*
        final Object o = m_outputs.get( key );
        if( null == o )
        { m_outputs.put( key, stream ); }
        else if( o instanceof ArrayList )
        { ( (ArrayList)o ).add( stream ); }

        else

        { final ArrayList list = new ArrayList(); list.add( o ); list.add( stream ); m_outputs.put( key, list ); }

        */
        return stream;
        }
        catch( final IOException ioe )

        { final String message = "Exception caught while storing a stream "; getLogger().warn( message, ioe ); throw new RuntimeException( message + ": " + ioe ); }

        }

        public synchronized void remove( final String key )
        {
        /*
        Object o = m_inputs.remove( key );
        if( null != o )
        {
        if( o instanceof InputStream )

        { IOUtil.shutdownStream( (InputStream)o ); }

        else
        {
        final ArrayList list = (ArrayList)o;
        final int size = list.size();

        for( int i = 0; i < size; i++ )

        { IOUtil.shutdownStream( (InputStream)list.get( i ) ); }

        }
        }

        o = m_outputs.remove( key );
        if( null != o )
        {
        if( o instanceof OutputStream )

        { IOUtil.shutdownStream( (OutputStream)o ); }

        else
        {
        final ArrayList list = (ArrayList)o;
        final int size = list.size();

        for( int i = 0; i < size; i++ )

        { IOUtil.shutdownStream( (OutputStream)list.get( 0 ) ); }

        }
        }
        */
        super.remove( key );
        }

        public long getSize(final String key) {
        try

        { return getFile(key).length(); }

        catch(IOException e)

        { return 0; }

        }
        }

        Show
        r_q_d Quande Ren added a comment - Hi, Stefano, Here is the changed version of File_Persistent_Object_Repository.java according to your suggestion No. 1. I tested the basic functions according to my knowledge and it is working. There's no memory leak anymore. Let me know your comments. Quande /*********************************************************************** Copyright (c) 2000-2004 The Apache Software Foundation. * All rights reserved. * ------------------------------------------------------------------- * Licensed under the Apache License, Version 2.0 (the "License"); you * may not use this file except in compliance with the License. You * may obtain a copy of the License at: * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or * implied. See the License for the specific language governing * permissions and limitations under the License. * ***********************************************************************/ package org.apache.james.mailrepository.filepair; import java.io.BufferedOutputStream; import java.io.IOException; import java.io.InputStream; import java.io.OutputStream; import java.util.ArrayList; import java.util.HashMap; import org.apache.avalon.cornerstone.services.store.StreamRepository; import org.apache.avalon.excalibur.io.IOUtil; /** Implementation of a StreamRepository to a File. TODO: -retieve(String key) should return a FilterInputStream to allow mark and reset methods. (working not like BufferedInputStream!!!) * */ public class File_Persistent_Stream_Repository extends AbstractFileRepository implements StreamRepository { //protected final HashMap m_inputs = new HashMap(); //protected final HashMap m_outputs = new HashMap(); protected String getExtensionDecorator() { return ".FileStreamStore"; } /** Get the object associated to the given unique key. */ public synchronized InputStream get( final String key ) { try { final ResettableFileInputStream stream = new ResettableFileInputStream( getFile( key ) ); /* final Object o = m_inputs.get( key ); if( null == o ) { m_inputs.put( key, stream ); } else if( o instanceof ArrayList ) { ( (ArrayList)o ).add( stream ); } else { final ArrayList list = new ArrayList(); list.add( o ); list.add( stream ); m_inputs.put( key, list ); } */ return stream; } catch( final IOException ioe ) { final String message = "Exception caught while retrieving a stream "; getLogger().warn( message, ioe ); throw new RuntimeException( message + ": " + ioe ); } } /** * Store the given object and associates it to the given key */ public synchronized OutputStream put( final String key ) { try { final OutputStream outputStream = getOutputStream( key ); final BufferedOutputStream stream = new BufferedOutputStream( outputStream ); /* final Object o = m_outputs.get( key ); if( null == o ) { m_outputs.put( key, stream ); } else if( o instanceof ArrayList ) { ( (ArrayList)o ).add( stream ); } else { final ArrayList list = new ArrayList(); list.add( o ); list.add( stream ); m_outputs.put( key, list ); } */ return stream; } catch( final IOException ioe ) { final String message = "Exception caught while storing a stream "; getLogger().warn( message, ioe ); throw new RuntimeException( message + ": " + ioe ); } } public synchronized void remove( final String key ) { /* Object o = m_inputs.remove( key ); if( null != o ) { if( o instanceof InputStream ) { IOUtil.shutdownStream( (InputStream)o ); } else { final ArrayList list = (ArrayList)o; final int size = list.size(); for( int i = 0; i < size; i++ ) { IOUtil.shutdownStream( (InputStream)list.get( i ) ); } } } o = m_outputs.remove( key ); if( null != o ) { if( o instanceof OutputStream ) { IOUtil.shutdownStream( (OutputStream)o ); } else { final ArrayList list = (ArrayList)o; final int size = list.size(); for( int i = 0; i < size; i++ ) { IOUtil.shutdownStream( (OutputStream)list.get( 0 ) ); } } } */ super.remove( key ); } public long getSize(final String key) { try { return getFile(key).length(); } catch(IOException e) { return 0; } } }
        Hide
        tlex Markus Kühn added a comment -

        It is File_Persistent_Stream_Repository.java and not File_Persistent_Object_Repository.java.

        Show
        tlex Markus Kühn added a comment - It is File_Persistent_Stream_Repository.java and not File_Persistent_Object_Repository.java.
        Hide
        brainlounge Bernd Fondermann added a comment -

        I can reproduce the bug under James 2.2.0 with the following Postage szenario:

        <scenario id="release_james512" runtimeMinutes="480"><!-- 8h=480, 1d=1440, 3d=4320 -->
        <users>
        <internal count="1" username-prefix="test_int" domain="mail.james.local" password="test" reuseExisting="yes" />
        <external count="0" username-prefix="test_ext" domain="mail.sample.com" />
        </users>
        <profiles>
        <profile name="ext-int" source="extern" target="intern">
        </profile>
        <profile name="int-ext" source="intern" target="extern">
        </profile>
        <profile name="int-int" source="intern" target="intern">
        <send count-per-min="25" subject="int2int" text-size-min="10" text-size-max="20" binary-size-min="0" binary-size-max="0" />
        <send count-per-min="25" subject="int2int" text-size-min="10" text-size-max="20" binary-size-min="0" binary-size-max="0" />
        <send count-per-min="25" subject="int2int" text-size-min="10" text-size-max="20" binary-size-min="0" binary-size-max="0" />
        <send count-per-min="25" subject="int2int" text-size-min="10" text-size-max="20" binary-size-min="0" binary-size-max="0" />
        <send count-per-min="25" subject="int2int" text-size-min="10" text-size-max="20" binary-size-min="0" binary-size-max="0" />
        <send count-per-min="25" subject="int2int" text-size-min="10" text-size-max="20" binary-size-min="0" binary-size-max="0" />
        <send count-per-min="25" subject="int2int" text-size-min="10" text-size-max="20" binary-size-min="0" binary-size-max="0" />
        <send count-per-min="25" subject="int2int" text-size-min="10" text-size-max="20" binary-size-min="0" binary-size-max="0" />
        <send count-per-min="25" subject="int2int" text-size-min="10" text-size-max="20" binary-size-min="0" binary-size-max="0" />
        <send count-per-min="25" subject="int2int" text-size-min="10" text-size-max="20" binary-size-min="0" binary-size-max="0" />

        <send count-per-min="25" subject="int2int" text-size-min="10" text-size-max="20" binary-size-min="0" binary-size-max="0" />
        <send count-per-min="25" subject="int2int" text-size-min="10" text-size-max="20" binary-size-min="0" binary-size-max="0" />
        <send count-per-min="25" subject="int2int" text-size-min="10" text-size-max="20" binary-size-min="0" binary-size-max="0" />
        <send count-per-min="25" subject="int2int" text-size-min="10" text-size-max="20" binary-size-min="0" binary-size-max="0" />
        <send count-per-min="25" subject="int2int" text-size-min="10" text-size-max="20" binary-size-min="0" binary-size-max="0" />
        <send count-per-min="25" subject="int2int" text-size-min="10" text-size-max="20" binary-size-min="0" binary-size-max="0" />
        <send count-per-min="25" subject="int2int" text-size-min="10" text-size-max="20" binary-size-min="0" binary-size-max="0" />
        <send count-per-min="25" subject="int2int" text-size-min="10" text-size-max="20" binary-size-min="0" binary-size-max="0" />
        <send count-per-min="25" subject="int2int" text-size-min="10" text-size-max="20" binary-size-min="0" binary-size-max="0" />
        <send count-per-min="25" subject="int2int" text-size-min="10" text-size-max="20" binary-size-min="0" binary-size-max="0" />

        </profile>
        </profiles>
        <testserver host="localhost">
        <smtp-forwarding port="2525" latecomer-wait-seconds="120"/>
        <smtp-inbound port="25" />
        <pop3 port="110" count-per-min="10" />
        <remotemanager port="4555" name="root" password="root" />
        <spam-account name="spam-sink" password="spam-sink" />
        <jvm-resources jmx-remoting-port="10201" />
        </testserver>
        </scenario>

        Show
        brainlounge Bernd Fondermann added a comment - I can reproduce the bug under James 2.2.0 with the following Postage szenario: <scenario id="release_james512" runtimeMinutes="480"><!-- 8h=480, 1d=1440, 3d=4320 --> <users> <internal count="1" username-prefix="test_int" domain="mail.james.local" password="test" reuseExisting="yes" /> <external count="0" username-prefix="test_ext" domain="mail.sample.com" /> </users> <profiles> <profile name="ext-int" source="extern" target="intern"> </profile> <profile name="int-ext" source="intern" target="extern"> </profile> <profile name="int-int" source="intern" target="intern"> <send count-per-min="25" subject="int2int" text-size-min="10" text-size-max="20" binary-size-min="0" binary-size-max="0" /> <send count-per-min="25" subject="int2int" text-size-min="10" text-size-max="20" binary-size-min="0" binary-size-max="0" /> <send count-per-min="25" subject="int2int" text-size-min="10" text-size-max="20" binary-size-min="0" binary-size-max="0" /> <send count-per-min="25" subject="int2int" text-size-min="10" text-size-max="20" binary-size-min="0" binary-size-max="0" /> <send count-per-min="25" subject="int2int" text-size-min="10" text-size-max="20" binary-size-min="0" binary-size-max="0" /> <send count-per-min="25" subject="int2int" text-size-min="10" text-size-max="20" binary-size-min="0" binary-size-max="0" /> <send count-per-min="25" subject="int2int" text-size-min="10" text-size-max="20" binary-size-min="0" binary-size-max="0" /> <send count-per-min="25" subject="int2int" text-size-min="10" text-size-max="20" binary-size-min="0" binary-size-max="0" /> <send count-per-min="25" subject="int2int" text-size-min="10" text-size-max="20" binary-size-min="0" binary-size-max="0" /> <send count-per-min="25" subject="int2int" text-size-min="10" text-size-max="20" binary-size-min="0" binary-size-max="0" /> <send count-per-min="25" subject="int2int" text-size-min="10" text-size-max="20" binary-size-min="0" binary-size-max="0" /> <send count-per-min="25" subject="int2int" text-size-min="10" text-size-max="20" binary-size-min="0" binary-size-max="0" /> <send count-per-min="25" subject="int2int" text-size-min="10" text-size-max="20" binary-size-min="0" binary-size-max="0" /> <send count-per-min="25" subject="int2int" text-size-min="10" text-size-max="20" binary-size-min="0" binary-size-max="0" /> <send count-per-min="25" subject="int2int" text-size-min="10" text-size-max="20" binary-size-min="0" binary-size-max="0" /> <send count-per-min="25" subject="int2int" text-size-min="10" text-size-max="20" binary-size-min="0" binary-size-max="0" /> <send count-per-min="25" subject="int2int" text-size-min="10" text-size-max="20" binary-size-min="0" binary-size-max="0" /> <send count-per-min="25" subject="int2int" text-size-min="10" text-size-max="20" binary-size-min="0" binary-size-max="0" /> <send count-per-min="25" subject="int2int" text-size-min="10" text-size-max="20" binary-size-min="0" binary-size-max="0" /> <send count-per-min="25" subject="int2int" text-size-min="10" text-size-max="20" binary-size-min="0" binary-size-max="0" /> </profile> </profiles> <testserver host="localhost"> <smtp-forwarding port="2525" latecomer-wait-seconds="120"/> <smtp-inbound port="25" /> <pop3 port="110" count-per-min="10" /> <remotemanager port="4555" name="root" password="root" /> <spam-account name="spam-sink" password="spam-sink" /> <jvm-resources jmx-remoting-port="10201" /> </testserver> </scenario>
        Hide
        brainlounge Bernd Fondermann added a comment -

        can reproduce against current branch v2.3

        Show
        brainlounge Bernd Fondermann added a comment - can reproduce against current branch v2.3
        Hide
        r_q_d Quande Ren added a comment -

        The changes I made causes another problem: When an email is send to a user, if the email specify "to", "cc", and "bcc" to the same user, instead of receiving one email, the user will receive three emails.

        I am not familiar with the email specification, I don't know if this is a correct behaviour. so this approach need to be reviewed.

        Show
        r_q_d Quande Ren added a comment - The changes I made causes another problem: When an email is send to a user, if the email specify "to", "cc", and "bcc" to the same user, instead of receiving one email, the user will receive three emails. I am not familiar with the email specification, I don't know if this is a correct behaviour. so this approach need to be reviewed.
        Hide
        brainlounge Bernd Fondermann added a comment -

        I can confirm that the suggested change (not caching the Streams) removes this acute memory leak. (there may be other leaks left.)
        the heap consumption is still at 30% after 50 minutes when it used to be over 100% after 30 minutes. and GC is still able to collect heap space.

        Show
        brainlounge Bernd Fondermann added a comment - I can confirm that the suggested change (not caching the Streams) removes this acute memory leak. (there may be other leaks left.) the heap consumption is still at 30% after 50 minutes when it used to be over 100% after 30 minutes. and GC is still able to collect heap space.
        Hide
        brainlounge Bernd Fondermann added a comment -

        shouldn't this

        for( int i = 0; i < size; i++ )

        { IOUtil.shutdownStream( (OutputStream)list.get( 0 ) ); }

        be

        for( int i = 0; i < size; i++ )

        { IOUtil.shutdownStream( (OutputStream)list.get( i ) ); }

        seems like this is not closing stream 1..(size-1)!

        Show
        brainlounge Bernd Fondermann added a comment - shouldn't this for( int i = 0; i < size; i++ ) { IOUtil.shutdownStream( (OutputStream)list.get( 0 ) ); } be for( int i = 0; i < size; i++ ) { IOUtil.shutdownStream( (OutputStream)list.get( i ) ); } seems like this is not closing stream 1..(size-1)!
        Hide
        brainlounge Bernd Fondermann added a comment -

        Quande, could you please re-test with this change (it's in line 156) and re-run the profiler? Thanks a lot!

        Show
        brainlounge Bernd Fondermann added a comment - Quande, could you please re-test with this change (it's in line 156) and re-run the profiler? Thanks a lot!
        Hide
        r_q_d Quande Ren added a comment -

        Hi, Bernd,

        I rollbacked to the original code and made the changes as you indicated, the memery still keeps growing up and not get gc'ed.

        The stream get closed immediately after been used. I don't know why it need to be cached. if it need to be cached, what event will trigger it to be removed from the HashMap?

        Quande

        Show
        r_q_d Quande Ren added a comment - Hi, Bernd, I rollbacked to the original code and made the changes as you indicated, the memery still keeps growing up and not get gc'ed. The stream get closed immediately after been used. I don't know why it need to be cached. if it need to be cached, what event will trigger it to be removed from the HashMap? Quande
        Hide
        r_q_d Quande Ren added a comment -

        BTW, The old code also sends three emails to a user when the user is specified in "to", "cc" and "bcc".

        Show
        r_q_d Quande Ren added a comment - BTW, The old code also sends three emails to a user when the user is specified in "to", "cc" and "bcc".
        Hide
        noel Noel J. Bergman added a comment -

        ...
        public synchronized InputStream get( final String key )
        {
        try
        {
        final ResettableFileInputStream stream =
        new ResettableFileInputStream( getFile( key ) );
        ...

        FWIW, that's old code. Stefano changed that in early March. r385090

        Show
        noel Noel J. Bergman added a comment - ... public synchronized InputStream get( final String key ) { try { final ResettableFileInputStream stream = new ResettableFileInputStream( getFile( key ) ); ... FWIW, that's old code. Stefano changed that in early March. r385090
        Hide
        bago Stefano Bagnara added a comment -

        Here is my proposal to remove RepositoryManager usage and to have fully weak references.
        Includes the fix for the memory leak in then File_Stream_Repositoy_Something....

        I would include this in trunk, test it and backport to 2.3 (part of assembly changes I did in past), but do as you like. You can use it for trunk, 3.0, 4.0, or even delete it

        Show
        bago Stefano Bagnara added a comment - Here is my proposal to remove RepositoryManager usage and to have fully weak references. Includes the fix for the memory leak in then File_Stream_Repositoy_Something.... I would include this in trunk, test it and backport to 2.3 (part of assembly changes I did in past), but do as you like. You can use it for trunk, 3.0, 4.0, or even delete it
        Hide
        brainlounge Bernd Fondermann added a comment -

        The patch fixed the worst problem. Instead of running into permanent unrecoverable OOMs after 30 minutes, it now lasts for around 100 minutes.
        In between though, we occaisonlly 'hit' the max heap ceiling. hitting the ceiling is accompanied by prompting logs like these:

        "mailstore: Exception retrieving mail: java.lang.RuntimeException: Exception caught while retrieving an object, cause: java.io.EOFException, so we're deleting it."

        they go away, as the heap recovers.
        the used heap starts to grow again, hits max, recovers. this all is repeated 2 or 3 times.
        finally, as we get
        "Exception in thread "RMI TCP Connection(341)-169.254.11.222" java.lang.OutOfMemoryError: Java heap space"
        James is unable to recover.

        note: my max heap is 64 MB.

        Show
        brainlounge Bernd Fondermann added a comment - The patch fixed the worst problem. Instead of running into permanent unrecoverable OOMs after 30 minutes, it now lasts for around 100 minutes. In between though, we occaisonlly 'hit' the max heap ceiling. hitting the ceiling is accompanied by prompting logs like these: "mailstore: Exception retrieving mail: java.lang.RuntimeException: Exception caught while retrieving an object, cause: java.io.EOFException, so we're deleting it." they go away, as the heap recovers. the used heap starts to grow again, hits max, recovers. this all is repeated 2 or 3 times. finally, as we get "Exception in thread "RMI TCP Connection(341)-169.254.11.222" java.lang.OutOfMemoryError: Java heap space" James is unable to recover. note: my max heap is 64 MB.
        Hide
        brainlounge Bernd Fondermann added a comment -

        the scenario is still not tested against non-file repositories. it is not so unlikely that this is a general problem.
        hope I can test with db/dbfile tomorrow.

        Show
        brainlounge Bernd Fondermann added a comment - the scenario is still not tested against non-file repositories. it is not so unlikely that this is a general problem. hope I can test with db/dbfile tomorrow.
        Hide
        brainlounge Bernd Fondermann added a comment -

        a db configuration runs into memory problems (Xmx=64m) after less than two hours, too.

        (this is a setup where 500 mails are created per minute, which is quite a load: 30 000 mails/hr!)

        Show
        brainlounge Bernd Fondermann added a comment - a db configuration runs into memory problems (Xmx=64m) after less than two hours, too. (this is a setup where 500 mails are created per minute, which is quite a load: 30 000 mails/hr!)
        Hide
        bago Stefano Bagnara added a comment -

        The specific issue has been resolved.
        If we can reproduce different OOM errors we should create new issues with specific information.

        Show
        bago Stefano Bagnara added a comment - The specific issue has been resolved. If we can reproduce different OOM errors we should create new issues with specific information.
        Hide
        danny@apache.org Danny Angus added a comment -

        Closing issue fixed in released version.

        Show
        danny@apache.org Danny Angus added a comment - Closing issue fixed in released version.

          People

          • Assignee:
            noel Noel J. Bergman
            Reporter:
            r_q_d Quande Ren
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development