SA Bugzilla – Bug 5941
parsing original SMTP Server not working properly
Last modified: 2008-10-07 06:45:19 UTC
Hello, I think I have hitten a bug in the way SA parses out the original SMTP host. I send and email from my mail client (to myself) through the SMTP server of GMX. SA thinks the Mail was sent directly from my computer (i.e. my dsl-routers IP) without using GMXs SMTP server. SPF_FAIL, RCVD_IN_PBL RBL, RCVD_IN_XBL RBL, RCVD_IN_SORBS_DUL and RDNS_DYNAMIC seem to think 85.55.41.198 was the SMTP server - which is wrong. 85.55.41.198 is the IP my dsl-router uses to connect to the Internet. The way of the mail is the following: MUA (kmail) -> GMXs SMTP Server -> GMX forwards it from anyaddress@gmx.net to seclinet@gmx.net (I have set it like this in my account preferences at GMX) -> fdm (which is a similar to fetchmail) fetches the mail via pop3 -> procmail (gets fed by fdm) -> spamassassin (called from procmail as first rule). Someone on the sa-users list pointed out that the following header might be the problem and that SA might not hanlde fdm. Received: by localhost (fdm 1.5, account "gmx"); Mon, 14 Jul 2008 01:04:12 +0200 Full headers: ------------------------------------------------------------------------------ Return-Path: <anyaddress@gmx.net> X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on momo.seclinet.org X-Spam-Level: **** X-Spam-Status: No, score=4.3 required=5.0 tests=AWL,BAYES_40,RCVD_IN_PBL,RCVD_IN_SORBS_DUL,RCVD_IN_XBL,RDNS_DYNAMIC,SPF_FAIL,TVD_SPACE_RATIO autolearn=no bayes=0.2760 language= report: * 0.9 RCVD_IN_PBL RBL: Received via a relay in Spamhaus PBL * [85.55.41.198 listed in zen.spamhaus.org] * 3.0 RCVD_IN_XBL RBL: Received via a relay in Spamhaus XBL * 0.9 RCVD_IN_SORBS_DUL RBL: SORBS: sent directly from dynamic IP address * [85.55.41.198 listed in dnsbl.sorbs.net] * 0.7 SPF_FAIL SPF: sender does not match SPF record (fail) * [SPF failed: Please see http://www.openspf.org/Why?s=mfrom&id=anyaddress%40gmx.net&ip=85.55.41.198&r=momo.seclinet.org] * -0.2 BAYES_40 BODY: Bayesian spam probability is 20 to 40% * [score: 0.2760] * 2.2 TVD_SPACE_RATIO BODY: TVD_SPACE_RATIO * 0.1 RDNS_DYNAMIC Delivered to trusted network by host with * dynamic-looking rDNS * -3.4 AWL AWL: From: address is in the auto white-list X-Flags: 0000 Delivered-To: GMX delivery to seclinet@gmx.net Received: by localhost (fdm 1.5, account "gmx"); Mon, 14 Jul 2008 01:04:12 +0200 Received: (qmail 6881 invoked by alias); 13 Jul 2008 22:29:06 -0000 Delivered-To: GMX delivery to anyaddress@gmx.net Received: (qmail invoked by alias); 13 Jul 2008 22:29:06 -0000 Received: from 198.pool85-55-41.dynamic.orange.es (EHLO [192.168.0.25]) [85.55.41.198] by mail.gmx.net (mp004) with SMTP; 14 Jul 2008 00:29:06 +0200 X-Authenticated: #8384405 X-Provags-ID: V01U2FsdGVkX1/KEJsVuZLKMG4BVaXLiJgyzPl76GsqwvYJeDn+q7 XuSbVqmMorwDIp From: Tom Fernandes <anyaddress@gmx.net> To: Tom Fernandes <anyaddress@gmx.net> Subject: test-procmail Date: Mon, 14 Jul 2008 00:29:04 +0200 User-Agent: KMail/1.9.9 MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200807140029.04272.anyaddress@gmx.net> X-FuHaFi: 0.00 X-GMX-Antivirus: 0 (no virus found) X-GMX-Antispam: -2 (not scanned, spam filter disabled) X-Resent-By: Forwarder <forwarder@gmx.net> X-Resent-For: anyaddress@gmx.net X-Resent-To: seclinet@gmx.net X-GMX-UID: /PQbLLcNa0AodebBJTAzUog3Njh6dE7a X-Length: 2321 X-UID: 1521 ------------------------------------------------------------------------------ thanks, Tom
> Received: by localhost (fdm 1.5, account "gmx"); > Mon, 14 Jul 2008 01:04:12 +0200 > Received: (qmail 6881 invoked by alias); 13 Jul 2008 22:29:06 -0000 > Delivered-To: GMX delivery to anyaddress@gmx.net > Received: (qmail invoked by alias); 13 Jul 2008 22:29:06 -0000 > Received: from 198.pool85-55-41.dynamic.orange.es (EHLO [192.168.0.25]) > [85.55.41.198] > by mail.gmx.net (mp004) with SMTP; 14 Jul 2008 00:29:06 +0200 The problem here is that there's no noticeable transfer from mail.gmx.net to localhost; there's no Received line containing an IP address etc. So it results in a single hop, like so: [26366] dbg: metadata: X-Spam-Relays-External: [ ip=85.55.41.198 rdns=198.pool85-55-41.dynamic.orange.es helo=!192.168.0.25! by=mail.gmx.net ident= envfrom= intl=0 id= auth= msa=0 ] for the dynamic-IP-to-GMX hop. I think the best fix for you is to simply trust the GMX mailhost, and the dynamic IP range that you are in; add them to internal_networks. That way, mail from your machine to your machine via GMX will always wind up trusted, which is what you're after. Alternatively deliver the mail locally instead of hopping via GMX (simply install an MTA like postfix). Or alternatively still, fix fdm to add a decent Received line ;)
(In reply to comment #1) [...] > Alternatively deliver the mail locally instead of hopping via GMX (simply > install an MTA like postfix). Or alternatively still, fix fdm to add a decent > Received line ;) > From looking at http://tools.ietf.org/html/rfc2821#section-4.4 I would say that fdm - to be rfc-conform - should add the "FROM" field to the "Received:" line - and then this problem should be solved - is that correct? If so I would forward this info to the fdm author. thanks, Tom
(In reply to comment #2) > (In reply to comment #1) > [...] > > > Alternatively deliver the mail locally instead of hopping via GMX (simply > > install an MTA like postfix). Or alternatively still, fix fdm to add a decent > > Received line ;) > > > > From looking at > > http://tools.ietf.org/html/rfc2821#section-4.4 > > I would say that fdm - to be rfc-conform - should add the "FROM" field to the > "Received:" line - and then this problem should be solved - is that correct? If it looked more like sendmail or Postfix' format, it would be best. ;)
Hi, > The problem here is that there's no noticeable transfer from mail.gmx.net to > localhost; there's no Received line containing an IP address etc. So it > results in a single hop, like so: > > [26366] dbg: metadata: X-Spam-Relays-External: [ ip=85.55.41.198 > rdns=198.pool85-55-41.dynamic.orange.es helo=!192.168.0.25! by=mail.gmx.net > ident= envfrom= intl=0 id= auth= msa=0 ] > > for the dynamic-IP-to-GMX hop. I doubt that this is really the problem. I did 2 checks: 1) I checked another mail which has also been send from a dynamic IP through a proper SMTP server (in this case web.de). This time SA doesn't think that the original SMTP server is the dynamic IP address it looks like (when looking at X-Spam-Status). fdm's "Received:" line is in both mails is exactly the same except for the timestamp. 2) I tested what happens if I remove the following lines (the ones added by fdm) --------------------------------------------------------- Received: by localhost (fdm 1.5, account "gmx"); Sat, 12 Jul 2008 ... --------------------------------------------------------- from the mail below and from the one in my first post and run them through SA again. Result ist that in the mail below the original SMTP gets found and in the other one SA still thinks the dynamic IP of the dsl-modem is the original SMTP server. For testing purpose I switched the original IP 85.55.41.198 (of the senders dsl-modem) with 81.41.72.156 (which is the one from the first post) in the source of the mail just to be sure that the different IP won't make a difference. I also substituted the original surname of the sender with xxx. Full headers: ------------------------------------------------------------------------- Return-Path: <assis.xxx@web.de> X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on momo.seclinet.org X-Spam-Level: X-Spam-Status: No, score=-3.6 required=5.0 tests=BAYES_00 autolearn=ham bayes=0.0000 language=de report: * -3.6 BAYES_00 BODY: Bayesian spam probability is 0 to 1% * [score: 0.0000] X-Flags: 1001 Delivered-To: GMX delivery to seclinet@gmx.net Received: by localhost (fdm 1.5, account "gmx"); Sat, 12 Jul 2008 18:07:54 +0000 Received: (qmail 19718 invoked by alias); 5 Jul 2008 21:12:56 -0000 Delivered-To: GMX delivery to anyaddress@gmx.net Received: (qmail invoked by alias); 05 Jul 2008 21:12:56 -0000 Received: from fmmailgate01.web.de (EHLO fmmailgate01.web.de) [217.72.192.221] by mx0.gmx.net (mx005) with SMTP; 05 Jul 2008 23:12:56 +0200 Received: from smtp08.web.de (fmsmtp08.dlan.cinetic.de [172.20.5.216]) by fmmailgate01.web.de (Postfix) with ESMTP id 5C447E69BF81 for <anyaddress@gmx.net>; Sat, 5 Jul 2008 23:12:56 +0200 (CEST) Received: from [85.55.41.198] (helo=[85.55.41.198]) by smtp08.web.de with asmtp (TLSv1:RC4-MD5:128) (WEB.DE 4.109 #226) id 1KFF3v-0003u5-00 for anyaddress@gmx.net; Sat, 05 Jul 2008 23:12:56 +0200 Subject: Re: Wegbeschreibung From: Assis <assis.xxx@web.de> To: Tom Fernandes <anyaddress@gmx.net> In-Reply-To: <200807051433.43420.anyaddress@gmx.net> References: <200807051433.43420.anyaddress@gmx.net> Content-Type: text/plain; charset=utf-8 Date: Sat, 05 Jul 2008 22:26:55 +0200 Message-Id: <1215289615.8598.13.camel@localhost.localdomain> Mime-Version: 1.0 X-Mailer: Evolution 2.10.3 Content-Transfer-Encoding: quoted-printable Sender: assis.xxx@web.de X-Sender: assis.xxx@web.de X-GMX-Antivirus: 0 (no virus found) X-GMX-Antispam: -2 (not scanned, spam filter disabled) X-Resent-By: Forwarder <forwarder@gmx.net> X-Resent-For: anyaddress@gmx.net X-Resent-To: seclinet@gmx.net X-GMX-UID: 4dtZAbI2QEVtHLCYOXRpsxJKNzg2NYJu X-Length: 2506 X-UID: 1518 -------------------------------------------------------------------------- thanks, Tom
I don't understand what you're saying. the fdm handover is invisible to SA -- that's the problem. Thinking about it more, I think the correct action for you to take is to trust the GMX relays and your own address in the dialup/dynamic address pool you're using, as I suggested. fundamentally, even if fdm fixes their Received lines, the issue will still be that the mail is originating from a dialup pool -- but the key fact is that it's *your* host in the dialup pool, and that needs to be trusted.
In a case like this, fixing fdm won't help. The Spamhaus hits are the very same when using fetchmail, for example, which is perfectly identified by SA. The issue is, that you indeed sent your mail via a single hop, aka direct MUA to MX. This is because your outgoing SMTP actually *is* the final recipients MX. You did send a mail from your dialup IP directly to the recipients MX. SA correctly checks that IP against the blacklists, and Spamhaus is punishing you... FWIW, I'm not convinced that trusting the entire Orange dialup IP space is a proper workaround. ;)
(In reply to comment #6) > FWIW, I'm not convinced that trusting the entire Orange dialup IP space is a > proper workaround. ;) actually, here's what might be better -- if you can login to the outbound relay using SMTP-AUTH SA should be able to detect that, and avoid applying those rules.
Hi, (In reply to comment #7) > actually, here's what might be better -- if you can login to the outbound relay > using SMTP-AUTH SA should be able to detect that, and avoid applying those > rules. Test case 1) - headers below Sent mail with MUA from dynamic IP with SMTP-AUTH through GMX's SMTP server to my own GMX address. Result: SA thinks the dynamic IP is the originating SMTP server Test case 2) - headers below Sent mail with MUA from dynamic IP with SMTP-AUTH through web.de's SMTP server to my own web.de address. Result: SA doesn't think the dynamic IP is the originating SMTP server Removing the "Received:" lines added by fdm didn't make any changes. I also went through some old mails that where send to me by people having an GMX account and using a MUA, and found out that in all of them SA confused the SMTP server to be dynamic IP address. Full headers: gmx headers ---------------------------------------------------------------- Return-Path: <seclinet@gmx.net> X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on momo.seclinet.org X-Spam-Level: * X-Spam-Status: No, score=1.6 required=5.0 tests=AWL,BAYES_05,RCVD_IN_PBL,RCVD_IN_SORBS_DUL,RDNS_DYNAMIC,SPF_FAIL autolearn=no bayes=0.0402 language= report: * 0.9 RCVD_IN_PBL RBL: Received via a relay in Spamhaus PBL * [85.55.47.166 listed in zen.spamhaus.org] * 0.9 RCVD_IN_SORBS_DUL RBL: SORBS: sent directly from dynamic IP address * [85.55.47.166 listed in dnsbl.sorbs.net] * 0.7 SPF_FAIL SPF: sender does not match SPF record (fail) * [SPF failed: Please see http://www.openspf.org/Why?s=mfrom&id=seclinet%40gmx.net&ip=85.55.47.166&r=momo.seclinet.org] * -1.1 BAYES_05 BODY: Bayesian spam probability is 1 to 5% * [score: 0.0402] * 0.1 RDNS_DYNAMIC Delivered to trusted network by host with * dynamic-looking rDNS * 0.1 AWL AWL: From: address is in the auto white-list X-Flags: 0000 Delivered-To: GMX delivery to seclinet@gmx.net Received: by localhost (fdm 1.5, account "gmx"); Thu, 17 Jul 2008 15:08:29 +0200 Received: (qmail invoked by alias); 17 Jul 2008 13:08:12 -0000 Received: from 166.pool85-55-47.dynamic.orange.es (EHLO [192.168.0.25]) [85.55.47.166] by mail.gmx.net (mp062) with SMTP; 17 Jul 2008 15:08:12 +0200 X-Authenticated: #22064775 X-Provags-ID: V01U2FsdGVkX19nTCR2S3ZnsGjQJEGZie3oRRwaX6FrXXjL3Di4py qYRO9qhe6E6KFo From: Tom Fernandes <seclinet@gmx.net> To: Tom Fernandes <seclinet@gmx.net> Subject: test from gmx to gmx Date: Thu, 17 Jul 2008 15:08:10 +0200 User-Agent: KMail/1.9.9 MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200807171508.10972.seclinet@gmx.net> X-FuHaFi: 0.00 X-GMX-Antivirus: 0 (no virus found) X-GMX-Antispam: -2 (not scanned, spam filter disabled) X-GMX-UID: D31McHM3MmApdezCO2BngwIxMjQ1N52M X-Length: 1949 X-UID: 1529 web.de headers -------------------------------------------------------- X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on momo.seclinet.org X-Spam-Level: X-Spam-Status: No, score=-5.4 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=unavailable bayes=0.0000 language= report: * -1.8 ALL_TRUSTED Passed through trusted hosts only via SMTP * -3.6 BAYES_00 BODY: Bayesian spam probability is 0 to 1% * [score: 0.0000] Received: by localhost (fdm 1.5, account "web"); Fri, 18 Jul 2008 12:58:22 +2400 Received: from [85.55.47.166] (helo=[192.168.0.25]) by smtp05.web.de with asmtp (TLSv1:AES256-SHA:256) (WEB.DE 4.109 #226) id 1KJSul-0002uZ-00 for carteras@web.de; Thu, 17 Jul 2008 14:48:55 +0200 From: Tom Fernandes <carteras@web.de> To: carteras@web.de Subject: test from web.de to web.de Date: Thu, 17 Jul 2008 14:48:44 +0200 User-Agent: KMail/1.9.9 MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200807171448.44996.carteras@web.de> Sender: carteras@web.de X-Sender: carteras@web.de X-Length: 1051 X-UID: 146 ---------------------------------------------------------------------- If this is really an SA problem I will look into SA code on the weekend and see if can come up with a patch. thanks so far, Tom
After reading in lib/Mail/SpamAssassin/Message/Metadata/Received.pm and doing some tests I think that the issue is that my SMTP-AUTH with the GMX SMTP server is not recognized by SA. AFAICT there is nothing in GMXs received headers where SA can tell from that I've been authenticating with GMX before sending my mail. It looks like GMX adds the "X-Authenticated" header for that but SA can't rely on that as it can be easily forged. When I modify GMXs received header in the mail manually (switch SMTP with ASMTP) and the according part in Received.pm (the line where GMXs mailserver is matched) so that SA thinks that I've been authenticated before relaying through GMX, the connection is trusted and the SPF and other rules are not run on the dynamic IP. It looks like this is a problem for all GMX users, using a pop fetcher + SA when receiving mails from somebody sending from a GMX account using a dialup IP. It does not make a difference if the sender uses SMTP or GMX webfrontend for sending. In both cases SA doesn't find out that the user authenticated before using GMX. Wouldn't it be possible to trust GMX server if it is the first hop in the received headers and it's not listed as an MX?. AFAICT GMX is known to authenticate all users who are relaying through them and the MX servers don't allow relaying (I did a fast telnet-check). If the headers of a mail send through the webfrontend is of any help - let me know. thanks, Tom
I just want to confirm the problem reported by Tom Fernandes. to summarize: SA believes that all e-mails send via GMX through a MUA (not the GMX web interface) are SPF_FAIL, because the original IP of the computer running the MUA differs from the allowed SPF range for GMX I am not sure whether this is an error in SA or an error in the internal email manipulation by GMX PS: I use SMTP authentication when connecting to the GMX SMTP server