James Server
  1. James Server
  2. JAMES-559

Message body get lost after call saveChanges() and move to other processor

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.3.0, 3.0.0
    • Fix Version/s: 2.3.0
    • Component/s: None
    • Labels:
      None

      Description

      After call saveChanges() in a mailet and move the mail to a other processor with ToProcessor the whole messageBody getting lost.

        Issue Links

          Activity

          Hide
          Norman Maurer added a comment -

          Noel wrote in mailinglist:

          OK, here is a scenario to reproduce. In the root processor, add:

          LogMessage
          ToRepository (pre-header)
          SetMimeHeader
          LogMessage
          ToRepository (post-header)
          ToProcessor (test)

          Needless to say, all of the ToRepository mailets should have passThrough
          set. And in the "test" processor, add:

          LogMessage
          ToRepository (post-processor)

          You could also instrument AvalonMailRepository.store to add
          mc.getMessage().writeTo(System.out) for debugging. You will see that the
          message is fine until we write to the spool, at which point it is corrupt.

          Show
          Norman Maurer added a comment - Noel wrote in mailinglist: OK, here is a scenario to reproduce. In the root processor, add: LogMessage ToRepository (pre-header) SetMimeHeader LogMessage ToRepository (post-header) ToProcessor (test) Needless to say, all of the ToRepository mailets should have passThrough set. And in the "test" processor, add: LogMessage ToRepository (post-processor) You could also instrument AvalonMailRepository.store to add mc.getMessage().writeTo(System.out) for debugging. You will see that the message is fine until we write to the spool, at which point it is corrupt.
          Hide
          Norman Maurer added a comment -

          After more testing it seems that the problem appears only with file repos. I tested it also with db and dbfile. No problem ..

          Show
          Norman Maurer added a comment - After more testing it seems that the problem appears only with file repos. I tested it also with db and dbfile. No problem ..
          Hide
          Noel J. Bergman added a comment -

          This appears to be a side-effect of optimizing message handling. Since the message is not in memory, we lose it when we call sr.put(key). See below.

          For v2.3, it might be best to revert behavior. For later versions, perhaps the file support from Jakarta Commons Transactions (http://jakarta.apache.org/commons/transaction/file/index.html) will resolve it.

          — Noel

          Index: src/java/org/apache/james/mailrepository/AvalonMailRepository.java
          ===================================================================
          — src/java/org/apache/james/mailrepository/AvalonMailRepository.java (revision 420194)
          +++ src/java/org/apache/james/mailrepository/AvalonMailRepository.java (working copy)
          @@ -236,6 +236,15 @@
          }
          }

          + private void logMessage(Mail mc, String label) throws java.io.IOException, MessagingException

          { + // NjB + System.out.println("---- AMR: store(" + destination + ") -----------"); + System.out.println("---- " + label + ") ----"); + mc.getMessage().writeTo(System.out); + System.out.println("-----------------------------"); + // NjB + }

          +
          /**

          • Stores a message in this repository. Shouldn't this return the key
          • under which it is stored?
            @@ -296,8 +305,11 @@
            if (saveStream) {
            OutputStream out = null;
            try { + logMessage(mc, "mark 4"); out = sr.put(key); + logMessage(mc, "mark 5"); mc.getMessage().writeTo(out); + logMessage(mc, "mark 6"); }

            finally

            { if (out != null) out.close(); }

          ------------------------------------------------------------------------------------------------------------

          ---- AMR: store(file://var/mail/spool/) -----------
          ---- mark 4) ----
          Return-Path: null
          Received: from localhost.localdomain ([127.0.0.1])
          by localhost.localdomain (JAMES SMTP Server 2.3.0b2) with SMTP ID 254
          for <me@localhost>;
          Sun, 9 Jul 2006 00:51:17 -0400 (EDT)
          Date: Now
          Subject: Binary search?
          To: Me
          From: Me
          X-MailetHeader: TheHeaderValue
          Content-Transfer-Encoding: 7bit
          Content-Type: text/plain; charset=us-ascii
          MIME-Version: 1.0
          Message-ID: <6044039.01152420680765.JavaMail.root@localhost.localdomain>

          Help me!

          Please.

          -----------------------------
          ---- AMR: store(file://var/mail/spool/) -----------
          ---- mark 5) ----
          Return-Path: null
          Received: from localhost.localdomain ([127.0.0.1])
          by localhost.localdomain (JAMES SMTP Server 2.3.0b2) with SMTP ID 254
          for <me@localhost>;
          Sun, 9 Jul 2006 00:51:17 -0400 (EDT)
          Date: Now
          Subject: Binary search?
          To: Me
          From: Me
          X-MailetHeader: TheHeaderValue
          Content-Transfer-Encoding: 7bit
          Content-Type: text/plain; charset=us-ascii
          MIME-Version: 1.0
          Message-ID: <6044039.01152420680765.JavaMail.root@localhost.localdomain>

          -----------------------------
          ---- AMR: store(file://var/mail/spool/) -----------
          ---- mark 6) ----
          Return-Path: null
          Received: from localhost.localdomain ([127.0.0.1])
          by localhost.localdomain (JAMES SMTP Server 2.3.0b2) with SMTP ID 254
          for <me@localhost>;
          Sun, 9 Jul 2006 00:51:17 -0400 (EDT)
          Date: Now
          Subject: Binary search?
          To: Me
          From: Me
          X-MailetHeader: TheHeaderValue
          Content-Transfer-Encoding: 7bit
          Content-Type: text/plain; charset=us-ascii
          MIME-Version: 1.0
          Message-ID: <6044039.01152420680765.JavaMail.root@localhost.localdomain>

          o: Me
          From: Me
          X-MailetHea-----------------------------

          Show
          Noel J. Bergman added a comment - This appears to be a side-effect of optimizing message handling. Since the message is not in memory, we lose it when we call sr.put(key). See below. For v2.3, it might be best to revert behavior. For later versions, perhaps the file support from Jakarta Commons Transactions ( http://jakarta.apache.org/commons/transaction/file/index.html ) will resolve it. — Noel Index: src/java/org/apache/james/mailrepository/AvalonMailRepository.java =================================================================== — src/java/org/apache/james/mailrepository/AvalonMailRepository.java (revision 420194) +++ src/java/org/apache/james/mailrepository/AvalonMailRepository.java (working copy) @@ -236,6 +236,15 @@ } } + private void logMessage(Mail mc, String label) throws java.io.IOException, MessagingException { + // NjB + System.out.println("---- AMR: store(" + destination + ") -----------"); + System.out.println("---- " + label + ") ----"); + mc.getMessage().writeTo(System.out); + System.out.println("-----------------------------"); + // NjB + } + /** Stores a message in this repository. Shouldn't this return the key under which it is stored? @@ -296,8 +305,11 @@ if (saveStream) { OutputStream out = null; try { + logMessage(mc, "mark 4"); out = sr.put(key); + logMessage(mc, "mark 5"); mc.getMessage().writeTo(out); + logMessage(mc, "mark 6"); } finally { if (out != null) out.close(); } ------------------------------------------------------------------------------------------------------------ ---- AMR: store( file://var/mail/spool/ ) ----------- ---- mark 4) ---- Return-Path: null Received: from localhost.localdomain ( [127.0.0.1] ) by localhost.localdomain (JAMES SMTP Server 2.3.0b2) with SMTP ID 254 for <me@localhost>; Sun, 9 Jul 2006 00:51:17 -0400 (EDT) Date: Now Subject: Binary search? To: Me From: Me X-MailetHeader: TheHeaderValue Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii MIME-Version: 1.0 Message-ID: <6044039.01152420680765.JavaMail.root@localhost.localdomain> Help me! Please. ----------------------------- ---- AMR: store( file://var/mail/spool/ ) ----------- ---- mark 5) ---- Return-Path: null Received: from localhost.localdomain ( [127.0.0.1] ) by localhost.localdomain (JAMES SMTP Server 2.3.0b2) with SMTP ID 254 for <me@localhost>; Sun, 9 Jul 2006 00:51:17 -0400 (EDT) Date: Now Subject: Binary search? To: Me From: Me X-MailetHeader: TheHeaderValue Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii MIME-Version: 1.0 Message-ID: <6044039.01152420680765.JavaMail.root@localhost.localdomain> ----------------------------- ---- AMR: store( file://var/mail/spool/ ) ----------- ---- mark 6) ---- Return-Path: null Received: from localhost.localdomain ( [127.0.0.1] ) by localhost.localdomain (JAMES SMTP Server 2.3.0b2) with SMTP ID 254 for <me@localhost>; Sun, 9 Jul 2006 00:51:17 -0400 (EDT) Date: Now Subject: Binary search? To: Me From: Me X-MailetHeader: TheHeaderValue Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii MIME-Version: 1.0 Message-ID: <6044039.01152420680765.JavaMail.root@localhost.localdomain> o: Me From: Me X-MailetHea-----------------------------
          Hide
          Stefano Bagnara added a comment -

          Status update: I wrote a unittest for this and the unittest fail. A good first step.

          The problem only happens in file repositories because with db repositories we don't stream from and to db, while with files we use streaming more aggressively.

          The problem probably arise from optimizations I added in january about loading only headers if only headers are changed and then write headers from memory and stream body from source if no body changes were done. When updating the same file there was a check that avoided to update the file when the whole message was not changed, but there is no "solution" for only headers changed.

          I'm currently investigating these 2 solutions:
          1) remove the "load only the headers" optimization and always load/parse the full message in memory
          2) change the file repository to write to a temporary file and rename it when the stream is closed.

          I already have found problems with both solutions, and that's why I don't have committed a patch yet....

          Show
          Stefano Bagnara added a comment - Status update: I wrote a unittest for this and the unittest fail. A good first step. The problem only happens in file repositories because with db repositories we don't stream from and to db, while with files we use streaming more aggressively. The problem probably arise from optimizations I added in january about loading only headers if only headers are changed and then write headers from memory and stream body from source if no body changes were done. When updating the same file there was a check that avoided to update the file when the whole message was not changed, but there is no "solution" for only headers changed. I'm currently investigating these 2 solutions: 1) remove the "load only the headers" optimization and always load/parse the full message in memory 2) change the file repository to write to a temporary file and rename it when the stream is closed. I already have found problems with both solutions, and that's why I don't have committed a patch yet....
          Hide
          Stefano Bagnara added a comment -

          Reverted the SharedFileInputStream to FileInputStream for file based repositories.
          This is a workaround that disable previous optimisation.

          There is a lot to do in 3.0 about this (both in file and db repositories).

          Show
          Stefano Bagnara added a comment - Reverted the SharedFileInputStream to FileInputStream for file based repositories. This is a workaround that disable previous optimisation. There is a lot to do in 3.0 about this (both in file and db repositories).
          Hide
          Vincenzo Gianferrari Pini added a comment -

          It works now for me. Thanks to all of the team for the work.
          I'll test 2.3.0b2 again in production on the weekend of July 22-23.

          Show
          Vincenzo Gianferrari Pini added a comment - It works now for me. Thanks to all of the team for the work. I'll test 2.3.0b2 again in production on the weekend of July 22-23.
          Hide
          Danny Angus added a comment -

          Closing issue fixed in released version.

          Show
          Danny Angus added a comment - Closing issue fixed in released version.

            People

            • Assignee:
              Stefano Bagnara
              Reporter:
              Norman Maurer
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development