Uploaded image for project: 'James Server'
  1. James Server
  2. JAMES-638

Wrong default encoding in text message (Autoconverted: from 8bit to quoted-printable)

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.3.0, 3.0.0
    • Fix Version/s: 2.3.0
    • Component/s: None
    • Labels:
      None
    • Environment:
      James 3.0, Linux

      Description

      Sending mail throught James to a second server (james as relay) make a wrong conversion of charset, I suppose that 8bit to quoted printable, the route of this mail is: sendmail(orig) -> james(relay) -> exchange(final)

      Input DATA (to James from sendmail):

      Date: Wed, 27 Sep 2006 01:05:07 +0200
      From: user@domain
      Message-Id: <200609262305.k8QN57L02881@domain>
      Subject: Jelou dolly

      2006-09-27 00:58:34 -> 2006-09-27 01:05:07
      mymoney=238
      yourmoney=162
      diff=31%(76)

      Received DATA (received in Exchange from James)

      Microsoft Mail Internet Headers Version 2.0
      Content-Transfer-Encoding: quoted-printable
      X-MIME-Autoconverted: from 8bit to quoted-printable by Apache JAMES
      Message-ID: <1979395.1691159311825941.JavaMail.root@domain>
      MIME-Version: 1.0
      Content-Type: text/plain; charset=us-ascii
      Date: Wed, 27 Sep 2006 01:05:07 +0200
      From: user@domain
      Subject: Jelou dolly
      Return-Path: user@domain
      X-OriginalArrivalTime: 26 Sep 2006 23:05:09.0546 (UTC) FILETIME=[36C490A0:01C6E1C0]

      2006-09-27 00:58:34 -> 2006-09-27 01:05:07
      mymoney#8
      yourmoney2
      diff%(76)

      If James is bypassed (usign IIS-SMTP-Relay instead of James), the body is OK, like this:

      Microsoft Mail Internet Headers Version 2.0
      content-class: urn:content-classes:message
      Date: Tue, 26 Sep 2006 01:02:04 +0200
      From: <user@domain>
      Message-ID: <200609252302.k8PN24A14699@domain>
      Subject: Jelou dolly
      X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1807
      Return-Path: <user@domain>
      X-OriginalArrivalTime: 25 Sep 2006 23:05:07.0593 (UTC) FILETIME=[0B30CF90:01C6E0F7]

      2006-09-26 00:48:13 -> 2006-09-26 01:02:04
      mymoney=120
      yourmoney=80
      diff=33%(40)

        Activity

        Hide
        bago Stefano Bagnara added a comment -

        Yes, this seems a bug. Not sure if the bug is in James or in Javamail.
        And this is probably related to JAMES-52.

        The problem here seems to be that the remote site does not support 8bitmime so James try to make sure the message is 7bit and call its own convertTo7bit. The convert to 7bit simply change the headers for the message to include quoted-printable. This was an hack for a bug in javamail and most time will not change the content because we temporarily disabled 8bitmime and messages received by james are not 8bit.

        The problem is that this message has no Content-Type/Content-Transfer-Encoding specification by default and adding quoted-printable "corrupt" the plain text body.

        This is related to this javamail bugs (I opened all of them):
        http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6456415
        http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6456422
        http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6456444

        At a glance we probably hit a bad/misbehaving part of javamail with this issue: I'm not sure wether we should try to find a fix/workaround now or try to wait for javamail 1.4.1!

        PS: this bug is probably also in 2.3.0!

        Show
        bago Stefano Bagnara added a comment - Yes, this seems a bug. Not sure if the bug is in James or in Javamail. And this is probably related to JAMES-52 . The problem here seems to be that the remote site does not support 8bitmime so James try to make sure the message is 7bit and call its own convertTo7bit. The convert to 7bit simply change the headers for the message to include quoted-printable. This was an hack for a bug in javamail and most time will not change the content because we temporarily disabled 8bitmime and messages received by james are not 8bit. The problem is that this message has no Content-Type/Content-Transfer-Encoding specification by default and adding quoted-printable "corrupt" the plain text body. This is related to this javamail bugs (I opened all of them): http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6456415 http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6456422 http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6456444 At a glance we probably hit a bad/misbehaving part of javamail with this issue: I'm not sure wether we should try to find a fix/workaround now or try to wait for javamail 1.4.1! PS: this bug is probably also in 2.3.0!
        Hide
        bago Stefano Bagnara added a comment -

        Temporarily added 2.3 as fix version: we have to confirm this happen in 2.3 and decide wether to try to workaround this for 2.3.0 final.

        Show
        bago Stefano Bagnara added a comment - Temporarily added 2.3 as fix version: we have to confirm this happen in 2.3 and decide wether to try to workaround this for 2.3.0 final.
        Hide
        technobcn Guillermo Grandes added a comment -

        More probes:

        The problem occurs when Exchange/Remote server do not have EHLO command (or is behind Cisco-PIX with "fixup protocol smtp" enabled, this mode mask the command EHLO and only permit HELO)

        Show
        technobcn Guillermo Grandes added a comment - More probes: The problem occurs when Exchange/Remote server do not have EHLO command (or is behind Cisco-PIX with "fixup protocol smtp" enabled, this mode mask the command EHLO and only permit HELO)
        Hide
        bago Stefano Bagnara added a comment -

        Of course because this is related to the remove server not supporting 8bitmime. So if the remote server does not support EHLO it also does not support 8BITMIME by default. If it supports EHLO and supports 8BITMIME then the message is sent as-is without conversions and is not corrupted.

        Show
        bago Stefano Bagnara added a comment - Of course because this is related to the remove server not supporting 8bitmime. So if the remote server does not support EHLO it also does not support 8BITMIME by default. If it supports EHLO and supports 8BITMIME then the message is sent as-is without conversions and is not corrupted.
        Hide
        technobcn Guillermo Grandes added a comment -

        I am watching in the RFC to find inspiration. Seemed it simpler
        http://www.faqs.org/rfcs/rfc2821.html (section 2.4)

        Show
        technobcn Guillermo Grandes added a comment - I am watching in the RFC to find inspiration. Seemed it simpler http://www.faqs.org/rfcs/rfc2821.html (section 2.4)
        Hide
        bago Stefano Bagnara added a comment -

        I don't understand your last comment. I know the rfc2821 and I re-read the p2.4 now but I can't understand what is simpler..

        We have to convert messages to 7 bit if they are 8bit and we send to 8 bit servers: the bug is that your message is not 8bit but james try anyway to convert it to 7bit and hit a known javamail bug.

        As a solution we could remove the convertTo7bit code because message we receive from the internet are not 8bit (because we don't currently support 8bitmime) so we can safely relay them to any server as-is (with no conversion).

        And I think this is a new bug introduced with 2.3.0: Is part of the code I wrote to support 8bitmime and I had to comment out most of it because of the 3 javamail bugs mentioned previously. James 2.2.0 had no "convertTo7bit" method in the remotedelivery.

        Show
        bago Stefano Bagnara added a comment - I don't understand your last comment. I know the rfc2821 and I re-read the p2.4 now but I can't understand what is simpler.. We have to convert messages to 7 bit if they are 8bit and we send to 8 bit servers: the bug is that your message is not 8bit but james try anyway to convert it to 7bit and hit a known javamail bug. As a solution we could remove the convertTo7bit code because message we receive from the internet are not 8bit (because we don't currently support 8bitmime) so we can safely relay them to any server as-is (with no conversion). And I think this is a new bug introduced with 2.3.0: Is part of the code I wrote to support 8bitmime and I had to comment out most of it because of the 3 javamail bugs mentioned previously. James 2.2.0 had no "convertTo7bit" method in the remotedelivery.
        Hide
        technobcn Guillermo Grandes added a comment -

        Excuse my english, must be a misspell my (I already know that you know the RFC, I am who I was reading it, to find a solution)... "Seemed it simple", it meant that I thought initially that it was easy to correct (to me) (I'm wrong).

        it has been happened to me that perhaps... in method "convertTo7bit"...

        private void convertTo7Bit(MimePart part) throws MessagingException, IOException {
        if (part.getHeader("MIME-Version") == null)

        { // or Content-Type or Content-Transfer-Encoding or Content-Disposition // i don't know who of them exactly... return; }

        // the rest of the code....
        }

        that you think?

        Show
        technobcn Guillermo Grandes added a comment - Excuse my english, must be a misspell my (I already know that you know the RFC, I am who I was reading it, to find a solution)... "Seemed it simple", it meant that I thought initially that it was easy to correct (to me) (I'm wrong). it has been happened to me that perhaps... in method "convertTo7bit"... private void convertTo7Bit(MimePart part) throws MessagingException, IOException { if (part.getHeader("MIME-Version") == null) { // or Content-Type or Content-Transfer-Encoding or Content-Disposition // i don't know who of them exactly... return; } // the rest of the code.... } that you think?
        Hide
        bago Stefano Bagnara added a comment -

        Yes, it should solve your problem.

        Btw maybe a better solution is to remove the whole call: that code hit a javamail bug, so if we really need to make a conversion we would corrupt the message because of the bug in javamail. So it is safe (imho) to remove the whole call. The conversion would be needed only for messages generated by James and not for messages received via SMTP.

        I'm sorry I have no time to investigate much more on this bug now, but my first attempt will be to remove that method at all (for 2.3.0). For trunk I have to think about it: I would like to find a solution that will work when javamail 1.4.1 will be released.

        Show
        bago Stefano Bagnara added a comment - Yes, it should solve your problem. Btw maybe a better solution is to remove the whole call: that code hit a javamail bug, so if we really need to make a conversion we would corrupt the message because of the bug in javamail. So it is safe (imho) to remove the whole call. The conversion would be needed only for messages generated by James and not for messages received via SMTP. I'm sorry I have no time to investigate much more on this bug now, but my first attempt will be to remove that method at all (for 2.3.0). For trunk I have to think about it: I would like to find a solution that will work when javamail 1.4.1 will be released.
        Hide
        norman Norman Maurer added a comment -

        I will take care (hopefully) tomorrow. I think we should remove the call completly

        Show
        norman Norman Maurer added a comment - I will take care (hopefully) tomorrow. I think we should remove the call completly
        Hide
        technobcn Guillermo Grandes added a comment -

        Good, I do not know this deeply, but if the problem is in the messages that come from another smtp, could be added an attribute to the messages which arrive from smtp (SMTPHandlerChain: SendMailHandler)...

        public void onMessage(SMTPSession session)

        { getLogger().debug("sending mail"); Mail mail = session.getMail(); + mail.setAttribute("REMOTEARRIBE"); // example // the rest of code // ... }

        and be verified it in the RemoteDelivery Mailet...

        • convertTo7Bit(message);
          + if (mail.setAttribute("REMOTEARRIBE") == null) convertTo7Bit(message);

        Besides to help with this bug, this attribute also would be useful to use with some Matcher to differentiate the locally generated messages of remote messages

        Show
        technobcn Guillermo Grandes added a comment - Good, I do not know this deeply, but if the problem is in the messages that come from another smtp, could be added an attribute to the messages which arrive from smtp (SMTPHandlerChain: SendMailHandler)... public void onMessage(SMTPSession session) { getLogger().debug("sending mail"); Mail mail = session.getMail(); + mail.setAttribute("REMOTEARRIBE"); // example // the rest of code // ... } and be verified it in the RemoteDelivery Mailet... convertTo7Bit(message); + if (mail.setAttribute("REMOTEARRIBE") == null) convertTo7Bit(message); Besides to help with this bug, this attribute also would be useful to use with some Matcher to differentiate the locally generated messages of remote messages
        Hide
        norman Norman Maurer added a comment -

        I hope i fixed it. Maybe you can test it by download the fixed version from:

        http://people.apache.org/~norman/james/downloads/james-638-fix/

        Any feedback whould be cool.

        Show
        norman Norman Maurer added a comment - I hope i fixed it. Maybe you can test it by download the fixed version from: http://people.apache.org/~norman/james/downloads/james-638-fix/ Any feedback whould be cool.
        Hide
        norman Norman Maurer added a comment -

        Move to rc4

        Show
        norman Norman Maurer added a comment - Move to rc4
        Hide
        norman Norman Maurer added a comment -

        Let us be optimistic

        Show
        norman Norman Maurer added a comment - Let us be optimistic
        Hide
        technobcn Guillermo Grandes added a comment -

        I'm usign the trunk-3.0. I'm to hardcore to use a stable 2.3 (joke)
        If possible I will test tomorrow the new trunk version with your corrections.

        Many Thanks.

        Show
        technobcn Guillermo Grandes added a comment - I'm usign the trunk-3.0. I'm to hardcore to use a stable 2.3 (joke) If possible I will test tomorrow the new trunk version with your corrections. Many Thanks.
        Hide
        technobcn Guillermo Grandes added a comment -

        well, already I have been able to prove, needed to have a version more or less stable (as I use trunk...) and re-enable the PIX "fixup protocol smtp"...

        All works good!

        Show
        technobcn Guillermo Grandes added a comment - well, already I have been able to prove, needed to have a version more or less stable (as I use trunk...) and re-enable the PIX "fixup protocol smtp"... All works good!
        Hide
        danny@apache.org Danny Angus added a comment -

        Closing issue fixed in released version.

        Show
        danny@apache.org Danny Angus added a comment - Closing issue fixed in released version.

          People

          • Assignee:
            norman Norman Maurer
            Reporter:
            technobcn Guillermo Grandes
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development