Message body get lost after call saveChanges() and move to other processor

• Key: JAMES-559
• Type: Bug
• Status: Closed
• Resolution: Fixed
• Priority: Blocker
• Assignee: Stefano Bagnara
• Reporter: Norman Maurer
• Affects Version/s: 2.3.0, 3.0
• Fix Version/s: 2.3.0

Description

After call saveChanges() in a mailet and move the mail to a other processor with ToProcessor the whole messageBody getting lost.

Comments

[Norman Maurer]

Noel wrote in mailinglist:

OK, here is a scenario to reproduce. In the root processor, add:

  LogMessage
  ToRepository (pre-header)
  SetMimeHeader
  LogMessage
  ToRepository (post-header)
  ToProcessor (test)

Needless to say, all of the ToRepository mailets should have passThrough
set. And in the "test" processor, add:

  LogMessage
  ToRepository (post-processor)

You could also instrument AvalonMailRepository.store to add
mc.getMessage().writeTo(System.out) for debugging. You will see that the
message is fine until we write to the spool, at which point it is corrupt.

[Norman Maurer]

After more testing it seems that the problem appears only with file repos. I tested it also with db and dbfile. No problem ..

[Noel J. Bergman]

This appears to be a side-effect of optimizing message handling. Since the message is not in memory, we lose it when we call sr.put(key). See below.

For v2.3, it might be best to revert behavior. For later versions, perhaps the file support from Jakarta Commons Transactions (http://jakarta.apache.org/commons/transaction/file/index.html) will resolve it.

    --- Noel

Index: src/java/org/apache/james/mailrepository/AvalonMailRepository.java
===================================================================
--- src/java/org/apache/james/mailrepository/AvalonMailRepository.java (revision 420194)
+++ src/java/org/apache/james/mailrepository/AvalonMailRepository.java (working copy)
@@ -236,6 +236,15 @@
         }
     }

+ private void logMessage(Mail mc, String label) throws java.io.IOException, MessagingException {
+ // NjB
+ System.out.println("---- AMR: store(" + destination + ") -----------");
+ System.out.println("---- " + label + ") ----");
+ mc.getMessage().writeTo(System.out);
+ System.out.println("-----------------------------");
+ // NjB
+ }
+
     /**
      * Stores a message in this repository. Shouldn't this return the key
      * under which it is stored?
@@ -296,8 +305,11 @@
                 if (saveStream) {
                     OutputStream out = null;
                     try {
+ logMessage(mc, "mark 4");
                         out = sr.put(key);
+ logMessage(mc, "mark 5");
                         mc.getMessage().writeTo(out);
+ logMessage(mc, "mark 6");
                     } finally {
                         if (out != null) out.close();
                     }


------------------------------------------------------------------------------------------------------------


---- AMR: store(file://var/mail/spool/) -----------
---- mark 4) ----
Return-Path: null
Received: from localhost.localdomain ([127.0.0.1])
          by localhost.localdomain (JAMES SMTP Server 2.3.0b2) with SMTP ID 254
          for <me@localhost>;
          Sun, 9 Jul 2006 00:51:17 -0400 (EDT)
Date: Now
Subject: Binary search?
To: Me
From: Me
X-MailetHeader: TheHeaderValue
Content-Transfer-Encoding: 7bit
Content-Type: text/plain; charset=us-ascii
MIME-Version: 1.0
Message-ID: <[e-mail]>

Help me!

Please. :-)

-----------------------------
---- AMR: store(file://var/mail/spool/) -----------
---- mark 5) ----
Return-Path: null
Received: from localhost.localdomain ([127.0.0.1])
          by localhost.localdomain (JAMES SMTP Server 2.3.0b2) with SMTP ID 254
          for <me@localhost>;
          Sun, 9 Jul 2006 00:51:17 -0400 (EDT)
Date: Now
Subject: Binary search?
To: Me
From: Me
X-MailetHeader: TheHeaderValue
Content-Transfer-Encoding: 7bit
Content-Type: text/plain; charset=us-ascii
MIME-Version: 1.0
Message-ID: <[e-mail]>

-----------------------------
---- AMR: store(file://var/mail/spool/) -----------
---- mark 6) ----
Return-Path: null
Received: from localhost.localdomain ([127.0.0.1])
          by localhost.localdomain (JAMES SMTP Server 2.3.0b2) with SMTP ID 254
          for <me@localhost>;
          Sun, 9 Jul 2006 00:51:17 -0400 (EDT)
Date: Now
Subject: Binary search?
To: Me
From: Me
X-MailetHeader: TheHeaderValue
Content-Transfer-Encoding: 7bit
Content-Type: text/plain; charset=us-ascii
MIME-Version: 1.0
Message-ID: <[e-mail]>

o: Me
From: Me
X-MailetHea-----------------------------

[Stefano Bagnara]

Status update: I wrote a unittest for this and the unittest fail. A good first step.

The problem only happens in file repositories because with db repositories we don't stream from and to db, while with files we use streaming more aggressively.

The problem probably arise from optimizations I added in january about loading only headers if only headers are changed and then write headers from memory and stream body from source if no body changes were done. When updating the same file there was a check that avoided to update the file when the whole message was not changed, but there is no "solution" for only headers changed.

I'm currently investigating these 2 solutions:
1) remove the "load only the headers" optimization and always load/parse the full message in memory
2) change the file repository to write to a temporary file and rename it when the stream is closed.

I already have found problems with both solutions, and that's why I don't have committed a patch yet....

[Stefano Bagnara]

Reverted the SharedFileInputStream to FileInputStream for file based repositories.
This is a workaround that disable previous optimisation.

There is a lot to do in 3.0 about this (both in file and db repositories).

[Vincenzo Gianferrari Pini]

It works now for me. Thanks to all of the team for the work.
I'll test 2.3.0b2 again in production on the weekend of July 22-23.

[Danny Angus]

Closing issue fixed in released version.