FileStreamStore diskspace leak for removed messages in file based spool under windows

• Key: JAMES-584
• Type: Bug
• Status: Closed
• Resolution: Fixed
• Priority: Major
• Assignee: Stefano Bagnara
• Reporter: Stefano Bagnara
• Component/s: MailStore & MailRepository
• Fix Version/s: 2.3.0

Description

Iwasa Kazmi reported this on server-user:
-----
Hi,

I sent a message to a local user and the delivery was
done successfully, but a FileStreamStore file is still
in the `spool' directory.
Is the FileStreamStore file used ?

I'm using james-2.3.0rc1 on WinXP.

I did the following:

1. edit config.xml for disable server name detection.

      <servernames autodetect="false" autodetectIP="true">
         <servername>localhost</servername>
      </servernames>

2. add user `iwasa'.

3. send email to iwasa@localhost.

Thank you,
Iwasa Kazmi
------
Confirmed by me here:
------
FWIW I can reproduce the issue with current 2.3rc1 and current trunk.

Config file as default for 2.3, repositories changed to file in trunk
(we still have derby as default in trunk).

I connect to remote manager, create an user. Connect to smtp server and
send a message to that user.

The XXX.Repository.FileStreamStore file is stuck there, in the spool.
If I restart James it got cleaned by the cleaning procedure in
AvalonMailRepository.

This seems to be a bug. Every file that is spooled remain there until a
restart. This is diskspace leak.

I tried manually calling from all the places the
ContainerUtil.dispose(mail) and set mail to null to be sure that the
remove were called and I can see that remove is called twice on that
file but it is not removed. It seems that a stream is still opened when
we try to remove it and it's not removed.

I've not investigated more on this. I only use filerepository in a small
production server that I rarely check. Noel, Vincenzo, can you check
what does it happen in the long term? Maybe this is a test for
Postage... spool folder should increase it size in a linear way if this
bug is there.
------
This could be platform specific: I'm able to reproduce it under windows.

This would also make sense because in unix os you can remove opened
files and they are really removed when the file is closed while under
windows an opened file cannot be deleted.

Comments

[Stefano Bagnara]

Let's mark this fixed. Optimism never killed anyone ;-)

[Stefano Bagnara]

Iwasa Kazmi found another way to reproduce this issue:
----
I inserted Redirect mailet in the "transport" processor.
(see config.xml.diff)
It duplicates a message for local-user before the RemoteDelivery runs.
Then I sent a mail to the external server via james.
FileStreamStore file in the spool was not deleted.
----

[Stefano Bagnara]

Optimism didn't work yesterday... let's test it again...

The main problem was that we were duplicating a Mail object using "new MailImpl(Mail)" that create a CoW proxy for the shared MimeMessage and then we were calling setMessage that replaced the previous message with no care of its "unreferencing".
So, now, if we call setMessage and a message was already there then we dispose the previous one first, and then we set the new one.

I also added a few mail.dispose to mailets that created local duplicates of serviced Mail and were not disposing it (AbstractRedirect, DSNBounce....).

Doing all this stuff I also thought that if something gone wrong after the mail duplication and before the disposal we would have leaked, so I added a few try/finally code to make sure the dispose was called anyway.

[Danny Angus]

Closing issue fixed in released version.