Wrong default encoding in text message (Autoconverted: from 8bit to quoted-printable)

• Key: JAMES-638
• Type: Bug
• Status: Closed
• Resolution: Fixed
• Priority: Major
• Assignee: Norman Maurer
• Reporter: Guillermo Grandes
• Affects Version/s: 2.3.0, 3.0
• Fix Version/s: 2.3.0

Description

Sending mail throught James to a second server (james as relay) make a wrong conversion of charset, I suppose that 8bit to quoted printable, the route of this mail is: sendmail(orig) -> james(relay) -> exchange(final)

Input DATA (to James from sendmail):

Date: Wed, 27 Sep 2006 01:05:07 +0200
From: user@domain
Message-Id: <200609262305.k8QN57L02881@domain>
Subject: Jelou dolly

2006-09-27 00:58:34 -> 2006-09-27 01:05:07
mymoney=238
yourmoney=162
diff=31%(76)

Received DATA (received in Exchange from James)

Microsoft Mail Internet Headers Version 2.0
Content-Transfer-Encoding: quoted-printable
X-MIME-Autoconverted: from 8bit to quoted-printable by Apache JAMES
Message-ID: <1979395.1691159311825941.JavaMail.root@domain>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Date: Wed, 27 Sep 2006 01:05:07 +0200
From: user@domain
Subject: Jelou dolly
Return-Path: user@domain
X-OriginalArrivalTime: 26 Sep 2006 23:05:09.0546 (UTC) FILETIME=[36C490A0:01C6E1C0]

2006-09-27 00:58:34 -> 2006-09-27 01:05:07
mymoney#8
yourmoney2
diff%(76)

If James is bypassed (usign IIS-SMTP-Relay instead of James), the body is OK, like this:

Microsoft Mail Internet Headers Version 2.0
content-class: urn:content-classes:message
Date: Tue, 26 Sep 2006 01:02:04 +0200
From: <user@domain>
Message-ID: <200609252302.k8PN24A14699@domain>
Subject: Jelou dolly
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1807
Return-Path: <user@domain>
X-OriginalArrivalTime: 25 Sep 2006 23:05:07.0593 (UTC) FILETIME=[0B30CF90:01C6E0F7]

2006-09-26 00:48:13 -> 2006-09-26 01:02:04
mymoney=120
yourmoney=80
diff=33%(40)

Comments

[Stefano Bagnara]

Yes, this seems a bug. Not sure if the bug is in James or in Javamail.
And this is probably related to JAMES-52.

The problem here seems to be that the remote site does not support 8bitmime so James try to make sure the message is 7bit and call its own convertTo7bit. The convert to 7bit simply change the headers for the message to include quoted-printable. This was an hack for a bug in javamail and most time will not change the content because we temporarily disabled 8bitmime and messages received by james are not 8bit.

The problem is that this message has no Content-Type/Content-Transfer-Encoding specification by default and adding quoted-printable "corrupt" the plain text body.

This is related to this javamail bugs (I opened all of them):
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6456415
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6456422
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6456444

At a glance we probably hit a bad/misbehaving part of javamail with this issue: I'm not sure wether we should try to find a fix/workaround now or try to wait for javamail 1.4.1!

PS: this bug is probably also in 2.3.0!

[Stefano Bagnara]

Temporarily added 2.3 as fix version: we have to confirm this happen in 2.3 and decide wether to try to workaround this for 2.3.0 final.

[Guillermo Grandes]

More probes:

The problem occurs when Exchange/Remote server do not have EHLO command (or is behind Cisco-PIX with "fixup protocol smtp" enabled, this mode mask the command EHLO and only permit HELO)

[Stefano Bagnara]

Of course because this is related to the remove server not supporting 8bitmime. So if the remote server does not support EHLO it also does not support 8BITMIME by default. If it supports EHLO and supports 8BITMIME then the message is sent as-is without conversions and is not corrupted.

[Guillermo Grandes]

I am watching in the RFC to find inspiration. Seemed it simpler :-)
http://www.faqs.org/rfcs/rfc2821.html (section 2.4)

[Stefano Bagnara]

I don't understand your last comment. I know the rfc2821 and I re-read the p2.4 now but I can't understand what is simpler..

We have to convert messages to 7 bit if they are 8bit and we send to 8 bit servers: the bug is that your message is not 8bit but james try anyway to convert it to 7bit and hit a known javamail bug.

As a solution we could remove the convertTo7bit code because message we receive from the internet are not 8bit (because we don't currently support 8bitmime) so we can safely relay them to any server as-is (with no conversion).

And I think this is a new bug introduced with 2.3.0: Is part of the code I wrote to support 8bitmime and I had to comment out most of it because of the 3 javamail bugs mentioned previously. James 2.2.0 had no "convertTo7bit" method in the remotedelivery.

[Guillermo Grandes]

Excuse my english, must be a misspell my (I already know that you know the RFC, I am who I was reading it, to find a solution)... "Seemed it simple", it meant that I thought initially that it was easy to correct (to me) (I'm wrong).

it has been happened to me that perhaps... in method "convertTo7bit"...

private void convertTo7Bit(MimePart part) throws MessagingException, IOException {
    if (part.getHeader("MIME-Version") == null) {
        // or Content-Type or Content-Transfer-Encoding or Content-Disposition
        // i don't know who of them exactly...
        return;
    }
    // the rest of the code....
}

that you think?

[Stefano Bagnara]

Yes, it should solve your problem.

Btw maybe a better solution is to remove the whole call: that code hit a javamail bug, so if we really need to make a conversion we would corrupt the message because of the bug in javamail. So it is safe (imho) to remove the whole call. The conversion would be needed only for messages generated by James and not for messages received via SMTP.

I'm sorry I have no time to investigate much more on this bug now, but my first attempt will be to remove that method at all (for 2.3.0). For trunk I have to think about it: I would like to find a solution that will work when javamail 1.4.1 will be released.

[Norman Maurer]

I will take care (hopefully) tomorrow. I think we should remove the call completly

[Guillermo Grandes]

Good, I do not know this deeply, but if the problem is in the messages that come from another smtp, could be added an attribute to the messages which arrive from smtp (SMTPHandlerChain: SendMailHandler)...

    public void onMessage(SMTPSession session) {
        getLogger().debug("sending mail");

        Mail mail = session.getMail();
    + mail.setAttribute("REMOTEARRIBE"); // example
        // the rest of code
        // ...
    }

and be verified it in the RemoteDelivery Mailet...

    - convertTo7Bit(message);
    + if (mail.setAttribute("REMOTEARRIBE") == null) convertTo7Bit(message);

Besides to help with this bug, this attribute also would be useful to use with some Matcher to differentiate the locally generated messages of remote messages :-)

[Norman Maurer]

I hope i fixed it. Maybe you can test it by download the fixed version from:

http://people.apache.org/~norman/james/downloads/james-638-fix/

Any feedback whould be cool.

Move to rc4

[Norman Maurer]

Let us be optimistic

[Guillermo Grandes]

I'm usign the trunk-3.0. I'm to hardcore to use a stable 2.3 (joke) ;-)
If possible I will test tomorrow the new trunk version with your corrections.

Many Thanks.

[Guillermo Grandes]

well, already I have been able to prove, needed to have a version more or less stable (as I use trunk...) and re-enable the PIX "fixup protocol smtp"...

All works good! :-)

[Danny Angus]

Closing issue fixed in released version.