James Server
  1. James Server
  2. JAMES-134

Large emails in the spool cause SpoolManager to throw OutOfMemoryError

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0a3, 2.1, 2.1.3, 2.2.0
    • Fix Version/s: 3.0-beta3
    • Labels:
      None
    • Environment:
      Operating System: MacOS X
      Platform: Macintosh

      Description

      Steps to repro:

      1. Send yourself a very large email (16 megs works for me)
      2. check the SpoolManager log and see this over and over:

      ERROR spoolmanager: Exception in JamesSpoolManager.run null
      java.lang.OutOfMemoryError

      What makes this problem particularly bad is that the spoolmanager doesn't move on to other
      messages but keeps pegging the CPU trying to process this email. To fix it, I have to shut down
      james, delete the email files out of spool and restart.

      EXPECTED: email should spool to the user as expected.

      1. replace.sh
        0.3 kB
        OG
      2. JamesMimeMessage.java
        13 kB
        OG
      3. JamesMimeMessage.java
        13 kB
        OG
      4. JamesMimeMessage.java
        6 kB
        OG
      5. JamesMimeMessage.java
        3 kB
        OG
      6. TestMemRec.java
        0.9 kB
        OG

        Issue Links

          Activity

          Matt Bishop created issue -
          Hide
          Noel J. Bergman added a comment -

          Try limiting your message size and/or increase the heap made available to the
          JVM (-Xmx).

          > EXPECTED: email should spool to the user as expected.

          Good trick if there is no memory left.

          James can try to move the message to the ERROR spool. If an error occurs
          then, however, the message will be GHOSTed (discarded).

          Show
          Noel J. Bergman added a comment - Try limiting your message size and/or increase the heap made available to the JVM (-Xmx). > EXPECTED: email should spool to the user as expected. Good trick if there is no memory left. James can try to move the message to the ERROR spool. If an error occurs then, however, the message will be GHOSTed (discarded).
          Hide
          Matt Bishop added a comment -

          I'm surprised that James tries to read the whole thing into memory before processing it. I would
          think it would sniff the header and stream it to the right account.

          Show
          Matt Bishop added a comment - I'm surprised that James tries to read the whole thing into memory before processing it. I would think it would sniff the header and stream it to the right account.
          Hide
          Rich added a comment -

          If I recall my earlier experiments correctly, the MimeMessage(Session,
          InputStream) constructor is the culprit here. It reads (or attempts to read)
          the whole message into memory, before returning, thus limiting message size to
          what can be held in memory at once.

          In my mailserver development I get around this by not constructing a MimeMessage
          (can't see that I need one), writing the message body right into a file.

          Show
          Rich added a comment - If I recall my earlier experiments correctly, the MimeMessage(Session, InputStream) constructor is the culprit here. It reads (or attempts to read) the whole message into memory, before returning, thus limiting message size to what can be held in memory at once. In my mailserver development I get around this by not constructing a MimeMessage (can't see that I need one), writing the message body right into a file.
          Serge Knystautas made changes -
          Field Original Value New Value
          issue.field.bugzillaimportkey 23906 13466
          Hide
          Danny Angus added a comment -

          I think this may be related.

          Show
          Danny Angus added a comment - I think this may be related.
          Danny Angus made changes -
          Link This issue is duplicated by JAMES-241 [ JAMES-241 ]
          Noel J. Bergman made changes -
          Description Steps to repro:

          1. Send yourself a very large email (16 megs works for me)
          2. check the SpoolManager log and see this over and over:

          ERROR spoolmanager: Exception in JamesSpoolManager.run null
          java.lang.OutOfMemoryError

          What makes this problem particularly bad is that the spoolmanager doesn't move on to other
          messages but keeps pegging the CPU trying to process this email. To fix it, I have to shut down
          james, delete the email files out of spool and restart.

          EXPECTED: email should spool to the user as expected.
          Steps to repro:

          1. Send yourself a very large email (16 megs works for me)
          2. check the SpoolManager log and see this over and over:

          ERROR spoolmanager: Exception in JamesSpoolManager.run null
          java.lang.OutOfMemoryError

          What makes this problem particularly bad is that the spoolmanager doesn't move on to other
          messages but keeps pegging the CPU trying to process this email. To fix it, I have to shut down
          james, delete the email files out of spool and restart.

          EXPECTED: email should spool to the user as expected.
          Priority Blocker [ 1 ]
          Assignee James Developers Mailing List [ server-dev@james.apache.org ]
          Environment Operating System: MacOS X
          Platform: Macintosh
          Operating System: MacOS X
          Platform: Macintosh
          Noel J. Bergman made changes -
          Affects Version/s 2.1.3 [ 10599 ]
          Affects Version/s 2.2.0a18 [ 10620 ]
          Affects Version/s 2.0a3 [ 10425 ]
          Hide
          OG added a comment -

          Problem exist also with W2K and James 2.2.0

          I have inserted -Xmx1024M to the command line in run.bat

          I then sent myself a 40MB file, and it worked BUT in windows task manager I see that the memory usage of James has increased to 300MB. It is now one day later, and James hasn't yet released the RAM that it took. It only grew. James takes now 333,272K.

          Show
          OG added a comment - Problem exist also with W2K and James 2.2.0 I have inserted -Xmx1024M to the command line in run.bat I then sent myself a 40MB file, and it worked BUT in windows task manager I see that the memory usage of James has increased to 300MB. It is now one day later, and James hasn't yet released the RAM that it took. It only grew. James takes now 333,272K.
          Hide
          OG added a comment -

          I point out the major problems:

          A) A mail server shouldn't take so much RAM.
          It doesn't have to take RAM of the size of the messages that it delivers.
          Not to mention that what it really does is taking RAM of a size that is 7 times the
          message size.
          B) Since it took so much RAM, it should give it back once it don't need it anymore.
          James doesn't give back the RAM that it took.

          My Jave -version:
          -------------------------------
          java version "1.4.1_01"
          Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.1_01-b01)
          Java HotSpot(TM) Client VM (build 1.4.1_01-b01, mixed mode)
          -------------------------------

          Show
          OG added a comment - I point out the major problems: A) A mail server shouldn't take so much RAM. It doesn't have to take RAM of the size of the messages that it delivers. Not to mention that what it really does is taking RAM of a size that is 7 times the message size. B) Since it took so much RAM, it should give it back once it don't need it anymore. James doesn't give back the RAM that it took. My Jave -version: ------------------------------- java version "1.4.1_01" Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.1_01-b01) Java HotSpot(TM) Client VM (build 1.4.1_01-b01, mixed mode) -------------------------------
          Hide
          Stefano Bagnara added a comment -

          OG "James doesn't give back the RAM that it took.": i think that it is a normal behaviour for the JVM to ask for virtual ram to the OS and never release it. Virtual memory management in modern OS has been implemented in a way that allow this. Windows in particular provides many memory occupation "numbers": what are you looking to? Windows don't provide real RAM usage. It never talks about RAM, instead you will find various "memory" indicators. There are many technical articles on the web that explain how to read those indicators.

          "It doesn't have to take RAM of the size of the messages that it delivers.": i could agree, but I should say "id depends". If a matcher/mailet has to analyze the message content to manage it then the message will be possibly decoded and more things that will need memory. It is delegated to the OS and the JVM to use the disk when the memory is not enough: the OS does a great job and does not make sense to handle "direct to disk temporary operations" to avoid memory usage.

          Show
          Stefano Bagnara added a comment - OG "James doesn't give back the RAM that it took.": i think that it is a normal behaviour for the JVM to ask for virtual ram to the OS and never release it. Virtual memory management in modern OS has been implemented in a way that allow this. Windows in particular provides many memory occupation "numbers": what are you looking to? Windows don't provide real RAM usage. It never talks about RAM, instead you will find various "memory" indicators. There are many technical articles on the web that explain how to read those indicators. "It doesn't have to take RAM of the size of the messages that it delivers.": i could agree, but I should say "id depends". If a matcher/mailet has to analyze the message content to manage it then the message will be possibly decoded and more things that will need memory. It is delegated to the OS and the JVM to use the disk when the memory is not enough: the OS does a great job and does not make sense to handle "direct to disk temporary operations" to avoid memory usage.
          Hide
          OG added a comment -

          Stefano - It isn't normal for the JVM to take memory and not give it back. There is a garbage collector, and it should manage the size of the heap to a reasonable size. That it isn't doing so could mean that resources are not being released properly, or that the GC doesn't get into action, and it can be forced into action, and it should act if James is sitting for a day on 300MB and a few seconds of operation time.

          The column I looked at is 'Mem Usage', and I am aware of the difference between physical RAM and Virtual Memory. The JVM has a maximum memory limit, so it isn't possible to set the JVM to accept any message size, so lets be large I thought, and did:
          ----------------------
          C:\>java -Xmx10000M
          Invalid maximum heap size: -Xmx10000M
          Could not create the Java virtual machine.
          ----------------------

          That is just 10GB and the JVM refuse to work with that. Why shouldn't I be allowed to send a 10GB message?

          Also, please tell me, can the address space in a 32bit machine be over 2^32, which is 4GB?
          Also, if you enlarge the VM to such sizes all the resources of that VM would also grow. So now your page file would take 10GB of disk space because James "needs" the VM. In reality however you would need more, because for some reason for me James took 7 times the size of the message so if you get a 10GB message, you would need 70GB of VM disk space because of that message. Do you have it? What if that was a 20GB message?
          Also, I don't know how much space in the RAM the lookup tables take, but do you? Do you know if windows or UNIX allow you to enlarge the VM page/swap file to what ever you want?

          Also, can you tell me why on earth does a mailet/matcher need to have all the message in the memory at once? Does mailets or matchers go randomly over messages that they process?

          Show
          OG added a comment - Stefano - It isn't normal for the JVM to take memory and not give it back. There is a garbage collector, and it should manage the size of the heap to a reasonable size. That it isn't doing so could mean that resources are not being released properly, or that the GC doesn't get into action, and it can be forced into action, and it should act if James is sitting for a day on 300MB and a few seconds of operation time. The column I looked at is 'Mem Usage', and I am aware of the difference between physical RAM and Virtual Memory. The JVM has a maximum memory limit, so it isn't possible to set the JVM to accept any message size, so lets be large I thought, and did: ---------------------- C:\>java -Xmx10000M Invalid maximum heap size: -Xmx10000M Could not create the Java virtual machine. ---------------------- That is just 10GB and the JVM refuse to work with that. Why shouldn't I be allowed to send a 10GB message? Also, please tell me, can the address space in a 32bit machine be over 2^32, which is 4GB? Also, if you enlarge the VM to such sizes all the resources of that VM would also grow. So now your page file would take 10GB of disk space because James "needs" the VM. In reality however you would need more, because for some reason for me James took 7 times the size of the message so if you get a 10GB message, you would need 70GB of VM disk space because of that message. Do you have it? What if that was a 20GB message? Also, I don't know how much space in the RAM the lookup tables take, but do you? Do you know if windows or UNIX allow you to enlarge the VM page/swap file to what ever you want? Also, can you tell me why on earth does a mailet/matcher need to have all the message in the memory at once? Does mailets or matchers go randomly over messages that they process?
          Hide
          Danny Angus added a comment -

          This is not a blocker, James is largely at the mercy of JavaMail and JVM for this issue (see further comments) and there is an effective workaround (increase heap max size)

          Show
          Danny Angus added a comment - This is not a blocker, James is largely at the mercy of JavaMail and JVM for this issue (see further comments) and there is an effective workaround (increase heap max size)
          Danny Angus made changes -
          Priority Blocker [ 1 ] Major [ 3 ]
          Bugzilla Id 23906
          Hide
          Danny Angus added a comment -

          OG Wrote:
          "It isn't normal for the JVM to take memory and not give it back."

          Yes it is, the JVM will take up to -Xmx and not release it ever.

          "There is a garbage collector, and it should manage the size of the heap to a reasonable size."

          The garbage collection manages the heap contents and can trigger fresh allocations of OS memory for the heap to grow into, but cannot reduce the memory allocated to the heap by the JVM/OS

          "That it isn't doing so could mean that resources are not being released properly,"

          Not correct, if the amount of heap space in-use increased and never decreased this would be true, but you are talking about the size of the availbe memeory, not the heap contents.

          " or that the GC doesn't get into action, and it can be forced into action, "

          It cannot be forced, only suggested, and it is considered bad practice for an application to interfere with garbage collection.

          "and it should act if James is sitting for a day on 300MB and a few seconds of operation time."

          Garbage collection is only carried out when an allocation is requested from the heap which would cause one or more of the heap partitions (young, tenured, permanent) to exceed a threshold % of their current size. If this collection doesn't result in enough space free in the heap it will trigger an increase in partition or heap size up to the absolute maximum size of the partition or the total heap, whichever is reached first.

          James is multi threaded, and so will require heap space for all active processes.

          Mailets are allowed, expected even, to examine and modify the content of messages. Therfore it is not unreasonable to use JavaMail as the message model. The draw back is that MimeMessage will read and decode messages "eagerly", largely because JavaMail is primarily a client API and Sun have expressed the opinion that they would not consider altering JavaMail to better suit the requirements of server applications.

          The solution is for someone to create new JavaMail content handlers. this is not a trivial task, but is one that the James commiters have considered in the past.

          Show
          Danny Angus added a comment - OG Wrote: "It isn't normal for the JVM to take memory and not give it back." Yes it is, the JVM will take up to -Xmx and not release it ever. "There is a garbage collector, and it should manage the size of the heap to a reasonable size." The garbage collection manages the heap contents and can trigger fresh allocations of OS memory for the heap to grow into, but cannot reduce the memory allocated to the heap by the JVM/OS "That it isn't doing so could mean that resources are not being released properly," Not correct, if the amount of heap space in-use increased and never decreased this would be true, but you are talking about the size of the availbe memeory, not the heap contents. " or that the GC doesn't get into action, and it can be forced into action, " It cannot be forced, only suggested, and it is considered bad practice for an application to interfere with garbage collection. "and it should act if James is sitting for a day on 300MB and a few seconds of operation time." Garbage collection is only carried out when an allocation is requested from the heap which would cause one or more of the heap partitions (young, tenured, permanent) to exceed a threshold % of their current size. If this collection doesn't result in enough space free in the heap it will trigger an increase in partition or heap size up to the absolute maximum size of the partition or the total heap, whichever is reached first. James is multi threaded, and so will require heap space for all active processes. Mailets are allowed, expected even, to examine and modify the content of messages. Therfore it is not unreasonable to use JavaMail as the message model. The draw back is that MimeMessage will read and decode messages "eagerly", largely because JavaMail is primarily a client API and Sun have expressed the opinion that they would not consider altering JavaMail to better suit the requirements of server applications. The solution is for someone to create new JavaMail content handlers. this is not a trivial task, but is one that the James commiters have considered in the past.
          Hide
          OG added a comment -

          I don't know what is the official definition of a "blocker", but I also interpret the problem to be a blocker since it is possible that someone would send a big message, and what happens then is an OutOfMemoryException, James quit delivering the mail, and occupy 99% of the CPU usage, which disturbs other processes too. So because the mail is blocked, the computer hangs, and other processes get blocked by CPU starvation, I thought that it is a blocker.

          By the way, I tried to increase my VM page file maximum size limit, and the maximum size that W2K allowed me is 4095MB. The JVM however is not that generous. After I increased W2K page file limit to the max, I tried to max out the -Xmx option of java. The maximum -Xmx that java agreed to run was 1532MB ('java -Xmx1532M'). So now if someone would send me a 1533/7=219MB file attachment, I am doomed.

          I have done another experiment, and sent myself the 40MB message again, and followed the James 'Mem Usage' column to see what happens. It started on 300MB from the previous time, then it shrunk to about 50MB, and started to grow slowly over a couple of minutes until it was back at 300MB. I learn from this, that the JVM heap can shrink. I learn from this that James did something when it got the new message that it didn't do before, and that thing got the heap to shrink. I just wish that James would clean after itself when it is done, and not just before it start. And even more than this I wish that the programmers here wouldn't load the messages to the memory at once. Even an antivirus that scans very big files doesn't load them at once to the memory, so I don't see why a mailet needs to do that. OK, I understand that you are saying that MimeMessage does just that. Probably what happens, is that the message go through 7 mailet/matcher that load it to memory 7 times. What can we do?

          Show
          OG added a comment - I don't know what is the official definition of a "blocker", but I also interpret the problem to be a blocker since it is possible that someone would send a big message, and what happens then is an OutOfMemoryException, James quit delivering the mail, and occupy 99% of the CPU usage, which disturbs other processes too. So because the mail is blocked, the computer hangs, and other processes get blocked by CPU starvation, I thought that it is a blocker. By the way, I tried to increase my VM page file maximum size limit, and the maximum size that W2K allowed me is 4095MB. The JVM however is not that generous. After I increased W2K page file limit to the max, I tried to max out the -Xmx option of java. The maximum -Xmx that java agreed to run was 1532MB ('java -Xmx1532M'). So now if someone would send me a 1533/7=219MB file attachment, I am doomed. I have done another experiment, and sent myself the 40MB message again, and followed the James 'Mem Usage' column to see what happens. It started on 300MB from the previous time, then it shrunk to about 50MB, and started to grow slowly over a couple of minutes until it was back at 300MB. I learn from this, that the JVM heap can shrink. I learn from this that James did something when it got the new message that it didn't do before, and that thing got the heap to shrink. I just wish that James would clean after itself when it is done, and not just before it start. And even more than this I wish that the programmers here wouldn't load the messages to the memory at once. Even an antivirus that scans very big files doesn't load them at once to the memory, so I don't see why a mailet needs to do that. OK, I understand that you are saying that MimeMessage does just that. Probably what happens, is that the message go through 7 mailet/matcher that load it to memory 7 times. What can we do?
          Hide
          Stefano Bagnara added a comment -

          OG: you can (must) specify the maximum message size you will accept from james.

          Abount the memory "shrink", again: read a book about windows memory management. You don't know what you are talking about. Do a simple test: open an application that take a lot of memory, look at the task manager, minimize the application. You will see the memory that drastically drop from 200-300MB to 5-6MB: do you really think this mean the application now uses only 5-6MB and has "released" that memory? No.

          Please stop looking at Task Manager numbers or read something about memory usage indicators: e.g: http://www.windowsitpro.com/Windows/Article/ArticleID/41095/41095.html

          BTW: this is an issue, and we should discuss better behaviour, but this is not blocking: james works. I think that more than 99% of mailservers will not accept 1GB email.

          BTW2: it is possible that some corporate provides commercial javamail implementations with better memory management: look around.

          Show
          Stefano Bagnara added a comment - OG: you can (must) specify the maximum message size you will accept from james. Abount the memory "shrink", again: read a book about windows memory management. You don't know what you are talking about. Do a simple test: open an application that take a lot of memory, look at the task manager, minimize the application. You will see the memory that drastically drop from 200-300MB to 5-6MB: do you really think this mean the application now uses only 5-6MB and has "released" that memory? No. Please stop looking at Task Manager numbers or read something about memory usage indicators: e.g: http://www.windowsitpro.com/Windows/Article/ArticleID/41095/41095.html BTW: this is an issue, and we should discuss better behaviour, but this is not blocking: james works. I think that more than 99% of mailservers will not accept 1GB email. BTW2: it is possible that some corporate provides commercial javamail implementations with better memory management: look around.
          Hide
          OG added a comment -

          Hi Stefano Bagnara,

          For my amusement I did the test that you have suggested: I opened Mozilla which takes about 30MB, and minimized it. To my amazement the 'Task Manager'(Processes TAB,'Mem Usage' column) did shrink to 3MB. I maximized and 'Mem Usage' got to 13MB, lower than it's previous value. I did the test again with James running in a command line while delivering the 40MB message, and I monitored it's 'Mem Usage'. Amazingly each time I minimized it the value shrunk to around 1MB, and started to rise again. I minimized and maximized several times during that mail transition, and the results were similar.

          Consequentially, I thought to myself that something is fishy about it. I opened windows help and looked to see why this 'Mem Usage' column doesn't behave as I expect. It turned out that 'Mem Usage' represent the current working set. I guess that W2K removes a minimized process from the current working set. Anyway I found out that what I was looking for, i.e. the real amount of memory that the process takes in Virtual Memory is under the column called 'VM Size'. The James 'VM size' was 339MB, this column was not affected by minimizations and maximizations, and when I attached the same 40MB file twice to a single message, the 'VM size' rose to 643MB. This rise is also reflected by the Performance TAB 'Mem Usage' which was then at 1,015,920.

          I restarted James with -Xmx1532M, and now I will try to crush it with a message that has many attachments of the 40MB file. Seven attachments did the trick and I have got the famous:
          ------------------------------------------------------------
          Spool=org.apache.james.mailrepository.AvalonSpoolRepository
          21/04/05 17:59:29 ERROR spoolmanager: Exception in processor <transport>
          java.lang.OutOfMemoryError
          21/04/05 17:59:29 ERROR spoolmanager: An error occurred processing Mail1114097348464-7 through transport
          21/04/05 17:59:29 ERROR spoolmanager: Result was error
          -------------------------------------------------------------
          May I add that the computer responsiveness was degraded very much during, even with less attachments.

          It isn't right for any program to sit on such large amounts of unused memory, I thought to myself, and so I have written a small program to test your claim that memory isn't given back to the system by java. I attached that test file TestMemRec.java here. The test makes a big array, and then set it to null, and then call System.gc(). I ran this test with 'java -Xmx1000M'. That test showed that even though the resources were released the memory wasn't given back to the system. I couldn't believe it, so I searched google and found the post that suggested to use -Xincgc, which link I wrote below. I ran my test again now with -Xincgc, and now the memory was given back to the system. I ran the program with the command 'java -Xmx1000M -Xincgc -Xloggc:gc.txt TestMemRec'

          post link:
          http://groups-beta.google.com/group/comp.lang.java.programmer/browse_frm/thread/6d9e6ab8f027d31f/16017bc2833aa6d5?q=java+%22return+memory%22&rnum=3&hl=en#16017bc2833aa6d5

          BTW - I read the link that you posted to see what you are trying to tell me, but I knew the stuff that was written there, because I have learned it many years ago. I don't know what your point was, sorry.

          BTW2- After changing to -Xincgc the mail server still had the same problem with big message delivery i.e. OurOfMemoryError.

          BTW3- For me James failed on 7x40=280MB and succeeded on 6x40=240MB. It isn't close to a 1GB message. People should be told that James can not transfer a 280MB message, instead of letting them find it out the hard way, when James fail, don't give any failure notice message, and quit delivering any more mail. People should also be told to use the -Xincgc in order to make the JVM return the unused memory to the system.

          Show
          OG added a comment - Hi Stefano Bagnara, For my amusement I did the test that you have suggested: I opened Mozilla which takes about 30MB, and minimized it. To my amazement the 'Task Manager'(Processes TAB,'Mem Usage' column) did shrink to 3MB. I maximized and 'Mem Usage' got to 13MB, lower than it's previous value. I did the test again with James running in a command line while delivering the 40MB message, and I monitored it's 'Mem Usage'. Amazingly each time I minimized it the value shrunk to around 1MB, and started to rise again. I minimized and maximized several times during that mail transition, and the results were similar. Consequentially, I thought to myself that something is fishy about it. I opened windows help and looked to see why this 'Mem Usage' column doesn't behave as I expect. It turned out that 'Mem Usage' represent the current working set. I guess that W2K removes a minimized process from the current working set. Anyway I found out that what I was looking for, i.e. the real amount of memory that the process takes in Virtual Memory is under the column called 'VM Size'. The James 'VM size' was 339MB, this column was not affected by minimizations and maximizations, and when I attached the same 40MB file twice to a single message, the 'VM size' rose to 643MB. This rise is also reflected by the Performance TAB 'Mem Usage' which was then at 1,015,920. I restarted James with -Xmx1532M, and now I will try to crush it with a message that has many attachments of the 40MB file. Seven attachments did the trick and I have got the famous: ------------------------------------------------------------ Spool=org.apache.james.mailrepository.AvalonSpoolRepository 21/04/05 17:59:29 ERROR spoolmanager: Exception in processor <transport> java.lang.OutOfMemoryError 21/04/05 17:59:29 ERROR spoolmanager: An error occurred processing Mail1114097348464-7 through transport 21/04/05 17:59:29 ERROR spoolmanager: Result was error ------------------------------------------------------------- May I add that the computer responsiveness was degraded very much during, even with less attachments. It isn't right for any program to sit on such large amounts of unused memory, I thought to myself, and so I have written a small program to test your claim that memory isn't given back to the system by java. I attached that test file TestMemRec.java here. The test makes a big array, and then set it to null, and then call System.gc(). I ran this test with 'java -Xmx1000M'. That test showed that even though the resources were released the memory wasn't given back to the system. I couldn't believe it, so I searched google and found the post that suggested to use -Xincgc, which link I wrote below. I ran my test again now with -Xincgc, and now the memory was given back to the system. I ran the program with the command 'java -Xmx1000M -Xincgc -Xloggc:gc.txt TestMemRec' post link: http://groups-beta.google.com/group/comp.lang.java.programmer/browse_frm/thread/6d9e6ab8f027d31f/16017bc2833aa6d5?q=java+%22return+memory%22&rnum=3&hl=en#16017bc2833aa6d5 BTW - I read the link that you posted to see what you are trying to tell me, but I knew the stuff that was written there, because I have learned it many years ago. I don't know what your point was, sorry. BTW2- After changing to -Xincgc the mail server still had the same problem with big message delivery i.e. OurOfMemoryError. BTW3- For me James failed on 7x40=280MB and succeeded on 6x40=240MB. It isn't close to a 1GB message. People should be told that James can not transfer a 280MB message, instead of letting them find it out the hard way, when James fail, don't give any failure notice message, and quit delivering any more mail. People should also be told to use the -Xincgc in order to make the JVM return the unused memory to the system.
          Hide
          OG added a comment -

          compile, do:
          a) java -Xmx1000M -Xincgc TestMemRec
          b) java -Xmx1000M TestMemRec

          And watch the 'VM size' column in 'Task Manager' Processes tab.

          Show
          OG added a comment - compile, do: a) java -Xmx1000M -Xincgc TestMemRec b) java -Xmx1000M TestMemRec And watch the 'VM size' column in 'Task Manager' Processes tab.
          OG made changes -
          Attachment TestMemRec.java [ 19724 ]
          Hide
          Stefano Bagnara added a comment -

          OG: i'm not saying this is NOT an issue. I simply said that I would not consider this an Blocking issue.
          I think that "MAJOR" is already enough. I agree with you that messages should be handled better and so do james committers (proof: this bug is still there).

          Show
          Stefano Bagnara added a comment - OG: i'm not saying this is NOT an issue. I simply said that I would not consider this an Blocking issue. I think that "MAJOR" is already enough. I agree with you that messages should be handled better and so do james committers (proof: this bug is still there).
          Hide
          Vincenzo Gianferrari Pini added a comment -

          First of all, reasonable maximum message sizes for an SMTP server should be quite below the values we are talking about here: SMTP != FTP. IMHO talking about 1 GB message size (or even 100 MB) is out of reality (remember to add about 33% to the attachment size to get the real message size).

          Secondly, DON'T USE INCREMENTAL GARBAGE COLLECTION (-Xincgc) with James (at least with jre 1.4.2_07 under linux, and <protocol>file</protocol>), as the effect in a real system is to slowly increase the average used memory, with an increasing "sawtooth" pattern, that will get in few hours to an OutOfMemory error. And this happens with a maximum message size limit set to values around 40MB. Instead, without -Xincgc, the behaviour is just an increase up to a reasonable horizontal asymptote.

          This does not mean that there may not be a memory leak somewhere in James: I recall (not sure though) something said by Noel about that when using the file protocol.

          Show
          Vincenzo Gianferrari Pini added a comment - First of all, reasonable maximum message sizes for an SMTP server should be quite below the values we are talking about here: SMTP != FTP. IMHO talking about 1 GB message size (or even 100 MB) is out of reality (remember to add about 33% to the attachment size to get the real message size). Secondly, DON'T USE INCREMENTAL GARBAGE COLLECTION (-Xincgc) with James (at least with jre 1.4.2_07 under linux, and <protocol>file</protocol>), as the effect in a real system is to slowly increase the average used memory, with an increasing "sawtooth" pattern, that will get in few hours to an OutOfMemory error. And this happens with a maximum message size limit set to values around 40MB. Instead, without -Xincgc, the behaviour is just an increase up to a reasonable horizontal asymptote. This does not mean that there may not be a memory leak somewhere in James: I recall (not sure though) something said by Noel about that when using the file protocol.
          Hide
          OG added a comment -

          Hello again. I have downloaded the james source code and took a closer look at the MimeMessage which is the prime suspect here. However, it seems like the constructor "public MimeMessage(Session session, InputStream is)" calls "parse(is)" and there it checks if "is instanceof SharedInputStream" and only if not it reads the whole thing to the memory. So how it looks to me (which may be wrong because I don't know a lot beyond it), is that if "is" will be made to be a SharedInputStream before it is passed on to the constructor of the MimeMessage than MimeMessage would not read it to the byte array "content" and use it as is.

          So now I want to know if this is indeed the only place where such problems rise, and if finding all calls to that constructors and replacing "is" with a "new MySharedIS(is)" would resolve the issue.

          What do you think?

          Show
          OG added a comment - Hello again. I have downloaded the james source code and took a closer look at the MimeMessage which is the prime suspect here. However, it seems like the constructor "public MimeMessage(Session session, InputStream is)" calls "parse(is)" and there it checks if "is instanceof SharedInputStream" and only if not it reads the whole thing to the memory. So how it looks to me (which may be wrong because I don't know a lot beyond it), is that if "is" will be made to be a SharedInputStream before it is passed on to the constructor of the MimeMessage than MimeMessage would not read it to the byte array "content" and use it as is. So now I want to know if this is indeed the only place where such problems rise, and if finding all calls to that constructors and replacing "is" with a "new MySharedIS(is)" would resolve the issue. What do you think?
          Hide
          OG added a comment -

          I looked further at MimeMessage, and I saw that also the constructor "public MimeMessage(MimeMessage source)" read it all to RAM (by reading into a ByteArrayOutputStream, and then into a SharedByteArrayInputStream which is also based on an array). Fortunately, it seems like the variables that are manipulated
          in these constructors are protected, and therefor could be accessed from a subclass that is in a different package, which means that we can make a subclass called
          "JamesMimeMessage" and have it override these constructors, and re do them with a "JamesSharedInputStream". The "JamesSharedInputStream" can be based on a
          file and not on a huge array, and have maybe a small buffer for performance, and the stuff it needs in order to implement SharedInputStream. Then the MimeMessage instances could be replaced with the "JamesMimeMessage" instances, and this will solve the problem with MimeMessage.

          What do you think?

          Any other places have huge byte[]?

          Show
          OG added a comment - I looked further at MimeMessage, and I saw that also the constructor "public MimeMessage(MimeMessage source)" read it all to RAM (by reading into a ByteArrayOutputStream, and then into a SharedByteArrayInputStream which is also based on an array). Fortunately, it seems like the variables that are manipulated in these constructors are protected, and therefor could be accessed from a subclass that is in a different package, which means that we can make a subclass called "JamesMimeMessage" and have it override these constructors, and re do them with a "JamesSharedInputStream". The "JamesSharedInputStream" can be based on a file and not on a huge array, and have maybe a small buffer for performance, and the stuff it needs in order to implement SharedInputStream. Then the MimeMessage instances could be replaced with the "JamesMimeMessage" instances, and this will solve the problem with MimeMessage. What do you think? Any other places have huge byte[]?
          Hide
          Stefano Bagnara added a comment -

          Few months ago I replaced the InputStreams used by James in mimemessages constructors to SharedByteArrayInputStreams but It didn't solve the memory issue.
          Maybe you could elaborate more on your proposal and provide a test and a patch for the issue.

          Show
          Stefano Bagnara added a comment - Few months ago I replaced the InputStreams used by James in mimemessages constructors to SharedByteArrayInputStreams but It didn't solve the memory issue. Maybe you could elaborate more on your proposal and provide a test and a patch for the issue.
          Hide
          OG added a comment -

          Hi,

          This file is meant to show what I mean. It is a class called JamesMimeMessage which subclass MimeMessage and make it use a SharedFileInputStream (which has already existed), instead of the byte array that caused problems with big messages. The idea is
          to change James classes that use MimeMessage so that they will use this JamesMimeMessage instead.

          PS. Stefano, SharedByteArrayInputStreams uses a byte array under it which is in memory and therefor that couldn't have helped. Also,
          I am not sure, but I think that one can't change MimeMessage and publish it, because of the copy rights. For that reason I have suggested
          a subclass, since such a solution that may be publishable.

          Show
          OG added a comment - Hi, This file is meant to show what I mean. It is a class called JamesMimeMessage which subclass MimeMessage and make it use a SharedFileInputStream (which has already existed), instead of the byte array that caused problems with big messages. The idea is to change James classes that use MimeMessage so that they will use this JamesMimeMessage instead. PS. Stefano, SharedByteArrayInputStreams uses a byte array under it which is in memory and therefor that couldn't have helped. Also, I am not sure, but I think that one can't change MimeMessage and publish it, because of the copy rights. For that reason I have suggested a subclass, since such a solution that may be publishable.
          OG made changes -
          Attachment JamesMimeMessage.java [ 12321976 ]
          Hide
          OG added a comment -

          Hi All,

          The new file I have attached (Second today) is working

          I have compiled it, and ran some big test files through a test account in the new server. The server ran all the time with under 27 MB of virtual memory footprint from which under 10 MB were physical RAM usage. The big test files sizes were 30 MB and 187 MB.

          Hip Hip :)

          There are more details in the first comments of the file.

          Show
          OG added a comment - Hi All, The new file I have attached (Second today) is working I have compiled it, and ran some big test files through a test account in the new server. The server ran all the time with under 27 MB of virtual memory footprint from which under 10 MB were physical RAM usage. The big test files sizes were 30 MB and 187 MB. Hip Hip : ) There are more details in the first comments of the file.
          OG made changes -
          Attachment JamesMimeMessage.java [ 12321983 ]
          Stefano Bagnara made changes -
          Assignee Stefano Bagnara [ bago ]
          Hide
          liqun tang added a comment -

          Hello
          I have fixed this bug as you talk to do,
          but it can not delivery a mail with a large attachement.
          The message will be put into the VAR/ERROR dir,
          and then I can not fetch the mail.
          Can anyone tell me why?

          Show
          liqun tang added a comment - Hello I have fixed this bug as you talk to do, but it can not delivery a mail with a large attachement. The message will be put into the VAR/ERROR dir, and then I can not fetch the mail. Can anyone tell me why?
          Hide
          Marco Trevisan added a comment -

          Hello all,

          My 2 cents about the problem:

          First. JVM >= 1.5.0 have dynamic heap resizing: the problem about "memory releasing" can be solved by simply switching to JDK 1.5.0 or higher. It is still important to ensure the correct objects finalization in order for gc to do its work.

          Second. About OG's early comment "A) A mail server shouldn't take so much RAM. It doesn't have to take RAM of the size of the messages that it delivers.":
          let's look at the situation at a little higher level. In order to process a mail message a mail server (which should be capable of processing it in complex ways) may need to have the whole message in RAM, i.e. when processing algorythms are not sequential and performance is welcome.
          In my opinion the message body should be kept in a file (as Rich said) only if there is no need to process it here.

          Regards, Marco

          Show
          Marco Trevisan added a comment - Hello all, My 2 cents about the problem: First. JVM >= 1.5.0 have dynamic heap resizing: the problem about "memory releasing" can be solved by simply switching to JDK 1.5.0 or higher. It is still important to ensure the correct objects finalization in order for gc to do its work. Second. About OG's early comment "A) A mail server shouldn't take so much RAM. It doesn't have to take RAM of the size of the messages that it delivers.": let's look at the situation at a little higher level. In order to process a mail message a mail server (which should be capable of processing it in complex ways) may need to have the whole message in RAM, i.e. when processing algorythms are not sequential and performance is welcome. In my opinion the message body should be kept in a file (as Rich said) only if there is no need to process it here. Regards, Marco
          Alan Cabrera made changes -
          Link This issue is duplicated by JAMES-241 [ JAMES-241 ]
          Alan Cabrera made changes -
          Link This issue is duplicated by JAMES-241 [ JAMES-241 ]
          Stefano Bagnara made changes -
          Fix Version/s Trunk [ 12312135 ]
          Hide
          Stefano Bagnara added a comment -

          Removed myself as assignee as I'm not willing to work on this in the near future. In the past 2 years I've applied some patches to increase the probability to be able to stream only the message without full loading it in memory, but this still does not happen on every case.

          Show
          Stefano Bagnara added a comment - Removed myself as assignee as I'm not willing to work on this in the near future. In the past 2 years I've applied some patches to increase the probability to be able to stream only the message without full loading it in memory, but this still does not happen on every case.
          Stefano Bagnara made changes -
          Assignee Stefano Bagnara [ bago ]
          Hide
          OG added a comment -

          Hi,
          I still work with the 2.2 version I made for myself. I see there is james version 2.3 now, and I wonder if this bug still exist over there too. Thanks.

          Show
          OG added a comment - Hi, I still work with the 2.2 version I made for myself. I see there is james version 2.3 now, and I wonder if this bug still exist over there too. Thanks.
          Hide
          Stefano Bagnara added a comment -

          Currently there is not a complete solution, but the scenario is a bit improved.
          We implemented streaming on reading but streaming on writing is still experimental. In config.xml you can read this:
          -------------------
          <!-- Set the size threshold for in memory handling of storing operations -->
          <!-- Default is currently 409600000 due to a bug with mysql and binary stream -->
          <!-- currently under investigation. Please change this only if you know what -->
          <!-- you do, this is EXPERIMENTAL -->
          <!--
          <inMemorySizeLimit>4096</inMemorySizeLimit>
          -->
          -----------------
          You may want to try it with an updated mysql and the latest connector/j then let us know the result of your tests.

          Anyway, as I wrote in the previous comment, there are and there will be cases of messages fully loaded in memory.
          If you use 2.2 you are strongly encouraged to upgrade (even if this bug is not solved) because of the dozen of bug fixed since 2.2.0.

          Show
          Stefano Bagnara added a comment - Currently there is not a complete solution, but the scenario is a bit improved. We implemented streaming on reading but streaming on writing is still experimental. In config.xml you can read this: ------------------- <!-- Set the size threshold for in memory handling of storing operations --> <!-- Default is currently 409600000 due to a bug with mysql and binary stream --> <!-- currently under investigation. Please change this only if you know what --> <!-- you do, this is EXPERIMENTAL --> <!-- <inMemorySizeLimit>4096</inMemorySizeLimit> --> ----------------- You may want to try it with an updated mysql and the latest connector/j then let us know the result of your tests. Anyway, as I wrote in the previous comment, there are and there will be cases of messages fully loaded in memory. If you use 2.2 you are strongly encouraged to upgrade (even if this bug is not solved) because of the dozen of bug fixed since 2.2.0.
          Hide
          OG added a comment -

          I post here an update to src\java\org\apache\james\core\JavaMimeMessage.java because the last one had issues of not deleting some of the temp files on windows, and not closing filehandles on linux. This one has operated well on my server for about a week. Use at your own risk. It is still for james version 2.2

          PS. Stefano, I added here a counter to my class, as I have seen in one post of yours about the old SharedFileInputStream behaviour, so thank you for that post.

          Show
          OG added a comment - I post here an update to src\java\org\apache\james\core\JavaMimeMessage.java because the last one had issues of not deleting some of the temp files on windows, and not closing filehandles on linux. This one has operated well on my server for about a week. Use at your own risk. It is still for james version 2.2 PS. Stefano, I added here a counter to my class, as I have seen in one post of yours about the old SharedFileInputStream behaviour, so thank you for that post.
          OG made changes -
          Attachment JamesMimeMessage.java [ 12382243 ]
          OG made changes -
          Attachment JamesMimeMessage.java [ 12397825 ]
          OG made changes -
          Attachment replace.sh [ 12397826 ]
          Hide
          OG added a comment - - edited

          Hi.

          I have attached two files today.

          1. A script that replace MimeMessage with JamesMimeMessage in src folder, and its subfolders.
          it is placed at james2.3.1/replace.sh.

          2. A version of the file JamesMimeMessage.java for James version 2.3.1.
          It is placed at james2.3.1/src/java/org/apache/james/core/JamesMimeMessage.java

          Put replace.sh at its place (src folder and replace.sh should be at the same folder),
          then run the ./replace.sh script, then put JamesMimeMessage at its place, and then build james.

          That is all.

          Note #1 that it is working in my server for about a month now, and
          Note #2 that I didn't test to much the replace.sh script, so backup
          your source before, and if it work well or not let me know, so I could
          delete this warning or fix the script.
          Note#3 put JamesMimeMessage at its place after running replace.sh, so
          that it will not change things in JamesMimeMessage.
          Note#4 If you add mailets or matchers in the future, run replace.sh on
          them too.

          Show
          OG added a comment - - edited Hi. I have attached two files today. 1. A script that replace MimeMessage with JamesMimeMessage in src folder, and its subfolders. it is placed at james2.3.1/replace.sh. 2. A version of the file JamesMimeMessage.java for James version 2.3.1. It is placed at james2.3.1/src/java/org/apache/james/core/JamesMimeMessage.java Put replace.sh at its place (src folder and replace.sh should be at the same folder), then run the ./replace.sh script, then put JamesMimeMessage at its place, and then build james. That is all. Note #1 that it is working in my server for about a month now, and Note #2 that I didn't test to much the replace.sh script, so backup your source before, and if it work well or not let me know, so I could delete this warning or fix the script. Note#3 put JamesMimeMessage at its place after running replace.sh, so that it will not change things in JamesMimeMessage. Note#4 If you add mailets or matchers in the future, run replace.sh on them too.
          Hide
          Norman Maurer added a comment -

          I think we optimised this stuff as far as possible. With the current implementation the message content will only get loaded into memory if you change the body of the message. So I think we can close this now

          Show
          Norman Maurer added a comment - I think we optimised this stuff as far as possible. With the current implementation the message content will only get loaded into memory if you change the body of the message. So I think we can close this now
          Norman Maurer made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Assignee Norman Maurer [ norman ]
          Fix Version/s 3.0-M3 [ 12315512 ]
          Fix Version/s Trunk [ 12312135 ]
          Resolution Fixed [ 1 ]
          Mark Thomas made changes -
          Workflow jira [ 23461 ] Default workflow, editable Closed status [ 12566640 ]
          Mark Thomas made changes -
          Workflow Default workflow, editable Closed status [ 12566640 ] jira [ 12582182 ]
          Norman Maurer made changes -
          Fix Version/s 3.0-beta2 [ 12316850 ]
          Fix Version/s 3.0-beta1 [ 12315512 ]

            People

            • Assignee:
              Norman Maurer
              Reporter:
              Matt Bishop
            • Votes:
              2 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development