SA Bugzilla – Bug 5856
X-Spam-Relays-Untrusted vs External
Last modified: 2009-04-22 08:31:45 UTC
All rules in dynrdns and helo tests are made with X-Spam-Relays-Untrusted, why? They should use X-Spam-Relays-External. Isn't the point to check direct-to-MX clients? As documentation and wiki says: "(internal_networks) This value is used when checking 'dial-up' or dynamic IP address blocklists, in order to detect direct-to-MX spamming." "Also worth noting: it's common for the "trusted" networks to extend further than the "internal" networks. If you are writing rules to match the host which delivered a mail into the SMTP MX server, you should use "external" instead of "untrusted", since it's common for "good" third-party senders to be put into the "trusted" list. (This is especially important for rules that match features of dynamic host senders, such as rDNS patterns etc.)" Yes, my trusted networks extends more than my internal network (which just contains my MXs). I am trusting many additional big ISP servers. Dynamic ISP users use those as relays, they should not be penaltized. I know untrusted/external is confusing for many users, but can't even developers get it straight? ;-)
Any comments? For now I have a workaround to "s/Untrusted/External/". Hopefully it doesn't need to be a permanent solution.
'I know untrusted/external is confusing for many users, but can't even developers get it straight? ;-)' no. paging Daryl to the discussion ;) I'm not sure about this. I can see it being useful to whitelist a frequent correspondent on a dynamic IP, for example, so that their mail doesn't hit RDNS_DYNAMIC. for this, I'd use "trusted_networks", rather than "internal_networks", since they're not 'relay hosts on [a] network considered to be MXes for your domain(s), or internal relays.'. the same would apply for HELO_DYNAMIC_IPADDR. Also, if I receive forwarded mail via my account at isp.net, I can add mx.isp.net to trusted_networks, and HELO_DYNAMIC_* and RDNS_DYNAMIC will then still work for me. And authenticated submissions to msa.isp.net will be ok, too, since the authentication tokens in the Received headers will be honoured, allowing the trusted-network boundary to encompass the authenticated machine.
If you are suggesting not to use External so you can whitelist "dynamic" IPs, I think that's wrong. If it's a "dynamic-looking" non-changing IP, then it should be whitelisted by other means. A new feature or a fix to the rule matching it, also update to documentation for that specific scenario. I don't think it's right to fix correctly working rules by making a documented feature working incorrectly.
Which do you think is more common? 1) Whitelisting dynamic-looking IPs 2) Whitelisting a lot of big ISP servers, so you save RBL checks. They also sometimes get on the lists, as they are big and some spam go through. And which do you think is more into the spirit of the features/documentation? ;) I think I saw some other bug that suggested another level of *_networks. Probably that would be the best way to go with specific whitelisting.
I forgot that in any case the "trusted" dynamic IP would appear as ALL_TRUSTED. I think that would handle it too without any extra new features. Just shortcircuit it, if you don't want any rules to hit.
I've been thinking more.. internal_networks should be updated to mean something like "hosts that relay mail from unknown third parties". I don't think there is any reason to limit it to "internal MXs"? Or is there something that requires _exactly_ this knowledge? This means you can include mailinglist servers there etc, that may relay directly from dynamic IPs. But you should not list big ISP servers, since they relay mail from their known customers, that would be under trusted_networks like usual. Please discuss.
(In reply to comment #0) > All rules in dynrdns and helo tests are made with X-Spam-Relays-Untrusted, why? In a time not long ago X-Spam-Relays-Untrusted was the only thing available which is why (almost?) all of the rules use it. I changed over some of the DNSBL tests to be "external aware" but I don't believe I changed any of the header rules. > They should use X-Spam-Relays-External. Isn't the point to check direct-to-MX > clients? In general, you are correct. As with the DNSBL rules, some exceptions may apply where X-Spam-Relays-Untrusted is actually more suited than X-Spam-Relays-External. > I know untrusted/external is confusing for many users, but can't even > developers get it straight? ;-) It's not so much that it was misunderstood, just that it wasn't done when the option became available to do so. The problem has long existed, but the option to be able to fix it (the addition of the X-Spam-Relays-External pseudo header) is rather new. (In reply to comment #1) > For now I have a workaround to "s/Untrusted/External/". Hopefully it doesn't > need to be a permanent solution. If you could create a patch (or just a cf file), and attach it to this bug, with the required changes and prefixing all the rule names with "T_BUG5868_", I would be more than happy to submit it for rule QA. Once it's verified that it's not going to cause any unwanted shifts in accuracy we can make the correction in the current rules. (In reply to comment #6) > I've been thinking more.. internal_networks should be updated to mean something > like "hosts that relay mail from unknown third parties". I don't think there is > any reason to limit it to "internal MXs"? Without purposely looking for "unknown third parties" (which can appear anywhere), yes, that is accurate. > This means you can include mailinglist servers there etc, that may relay > directly from dynamic IPs. Directly, no, that would not be wise as you'd hit on dynamic lists like crazy. > But you should not list big ISP servers, since they > relay mail from their known customers, that would be under trusted_networks > like usual. Big ISPs, sure. They're the most likely to have separated their MSAs and their outgoing MTAs. > Please discuss. It is my recommendation that you only trust (trusted_networks and internal_networks, or just trusted_networks for the same effect) your own relays (your MXes and anything after them) and any relays who routinely accept mail on your behalf and forward it. Almost never mailing lists (only do so if you can determine what their MX is and keep up to date with changes). The *only* time I would suggest extending trusted_networks (and NOT internal_networks) to relays that are not your MX (or after them) would be if you need to avoid DNSBL hits on someone you exchange mail with. I would never do it on the basis of performance reasons (DNS caching will take care of that).
Created attachment 4296 [details] T_BUG5868 test rules for Q/A run
(In reply to comment #8) > Created an attachment (id=4296) [details] > T_BUG5868 test rules for Q/A run thanks, added for testing: : jm 311...; svn commit -m "bug 5856: add test rules for QA, thanks to Henrik Krohns" rulesrc/sandbox/jm/20_bug5856.cf Adding rulesrc/sandbox/jm/20_bug5856.cf Transmitting file data . Committed revision 646718.
preliminary results -- Author: jm Files: rulesrc/sandbox/jm/20_bug5856.cf Log: bug 5856: add test rules for QA, thanks to Henrik Krohns Freqs for those rules in 'mc-fast' mass-check: MSECS SPAM% HAM% S/O RANK SCORE NAME 0 1998 1998 0.500 0.00 0.00 (all messages) 0.00000 50.0000 50.0000 0.500 0.00 0.00 (all messages as %) 0.06005 0.0000 0.0000 0.500 0.48 0.01 T_BUG5868_FAKE_HELO_MAIL_COM_DOM 0.04553 0.0501 0.0000 1.000 0.49 0.01 T_BUG5868_HELO_DYNAMIC_CHELLO_NL 0.08131 5.3554 0.0000 1.000 0.84 0.01 T_BUG5868_HELO_DYNAMIC_DHCP 0.07681 0.9510 0.0000 1.000 0.63 0.01 T_BUG5868_HELO_DYNAMIC_DIALIN 0.05654 3.7037 0.0000 1.000 0.81 0.01 T_BUG5868_HELO_DYNAMIC_HCC 0.03478 0.0501 0.0000 1.000 0.49 0.01 T_BUG5868_HELO_DYNAMIC_HEXIP 0.04453 0.0000 0.0000 0.500 0.48 0.01 T_BUG5868_HELO_DYNAMIC_HOME_NL 0.05854 11.1612 0.0000 1.000 0.93 0.01 T_BUG5868_HELO_DYNAMIC_IPADDR 0.06155 8.7087 0.0000 1.000 0.92 0.01 T_BUG5868_HELO_DYNAMIC_IPADDR2 0.05004 0.0000 0.0000 0.500 0.48 0.01 T_BUG5868_HELO_DYNAMIC_ROGERS 0.05054 5.3053 0.0000 1.000 0.84 0.01 T_BUG5868_HELO_DYNAMIC_SPLIT_IP 0.00000 26.6266 0.0000 1.000 0.98 0.01 T_BUG5868_RDNS_DYNAMIC 0.00000 49.0991 0.0000 1.000 1.00 0.01 T_BUG5868_RDNS_NONE More: http://ruleqa.spamassassin.org/20080410-r646718-b
ok, here's the results -- looks fine by me. http://ruleqa.spamassassin.org/?daterev=20080412-r647396-n&rule=%2F%28BUG5868%7CHELO_DYNAMIC%7CRDNS_DYNAMIC%7CFAKE_HELO%7CRDNS_NONE%29 0.00000 13.4610 0.0012 1.000 0.98 4.40 HELO_DYNAMIC_IPADDR 0.00000 13.4532 0.0012 1.000 0.98 0.01 T_BUG5868_HELO_DYNAMIC_IPADDR 0.00000 11.5054 0.0152 0.999 0.97 4.40 HELO_DYNAMIC_IPADDR2 0.00000 11.4985 0.0152 0.999 0.97 0.01 T_BUG5868_HELO_DYNAMIC_IPADDR2 0.00000 5.4785 0.0000 1.000 0.96 4.30 HELO_DYNAMIC_HCC 0.00000 5.4740 0.0000 1.000 0.96 0.01 T_BUG5868_HELO_DYNAMIC_HCC 0.00000 4.4332 0.0000 1.000 0.96 4.20 HELO_DYNAMIC_SPLIT_IP 0.00000 4.4286 0.0000 1.000 0.96 0.01 T_BUG5868_HELO_DYNAMIC_SPLIT_IP 0.00000 5.4623 0.0094 0.998 0.95 2.30 HELO_DYNAMIC_DHCP 0.00000 5.4596 0.0094 0.998 0.95 0.01 T_BUG5868_HELO_DYNAMIC_DHCP 0.00000 0.2808 0.0000 1.000 0.82 4.00 HELO_DYNAMIC_DIALIN 0.00000 0.2806 0.0000 1.000 0.81 0.01 T_BUG5868_HELO_DYNAMIC_DIALIN 0.00000 45.3012 0.4549 0.990 0.78 0.01 T_BUG5868_RDNS_DYNAMIC 0.00000 45.3253 0.4584 0.990 0.78 0.10 RDNS_DYNAMIC 0.00000 0.1172 0.0000 1.000 0.75 3.50 HELO_DYNAMIC_HOME_NL 0.00000 0.1171 0.0000 1.000 0.75 0.01 T_BUG5868_HELO_DYNAMIC_HOME_NL 0.00000 0.1097 0.0000 1.000 0.74 3.10 HELO_DYNAMIC_HEXIP 0.00000 0.1094 0.0000 1.000 0.74 0.01 T_BUG5868_HELO_DYNAMIC_HEXIP 0.00000 0.0925 0.0000 1.000 0.72 3.60 HELO_DYNAMIC_CHELLO_NL 0.00000 0.0924 0.0000 1.000 0.72 0.01 T_BUG5868_HELO_DYNAMIC_CHELLO_NL 0.00000 37.6698 2.8359 0.930 0.63 0.10 RDNS_NONE 0.00000 37.6463 2.8359 0.930 0.63 0.01 T_BUG5868_RDNS_NONE 0.00000 0.0102 0.0000 1.000 0.56 0.01 T_BUG5868_HELO_DYNAMIC_ROGERS 0.00000 0.0000 0.0000 0.500 0.50 0.00 HELO_DYNAMIC_ROGERS 0.00000 0.0203 0.0000 1.000 0.60 0.01 FAKE_HELO_MAIL_COM_DOM 0.00000 0.0203 0.0000 1.000 0.60 0.01 T_BUG5868_FAKE_HELO_MAIL_COM_DOM and the subrules: 0.00000 8.0770 0.0140 0.998 0.96 (n/a) __T_BUG5868_RDNS_DYNAMIC_HCC 0.00000 8.0819 0.0164 0.998 0.96 (n/a) __RDNS_DYNAMIC_HCC 0.00000 6.6430 0.0152 0.998 0.96 (n/a) __RDNS_DYNAMIC_SPLIT_IP 0.00000 6.6383 0.0187 0.997 0.95 (n/a) __T_BUG5868_RDNS_DYNAMIC_SPLIT_IP 0.00000 6.6439 0.0246 0.996 0.95 (n/a) __T_BUG5868_RDNS_INDICATOR_DYN 0.00000 1.8188 0.0000 1.000 0.92 (n/a) __T_BUG5868_RDNS_INDICATOR_TYPE2 0.00000 1.8126 0.0012 0.999 0.92 (n/a) __T_BUG5868_RDNS_INDICATOR_RES 0.00000 10.8432 0.0865 0.992 0.91 (n/a) __T_BUG5868_RDNS_INDICATOR_TYPE 0.00000 7.7726 0.0936 0.988 0.90 (n/a) __T_BUG5868_RDNS_DYNAMIC_DHCP 0.00000 7.7751 0.0959 0.988 0.89 (n/a) __RDNS_DYNAMIC_DHCP 0.00000 0.5587 0.0000 1.000 0.86 (n/a) __RDNS_DYNAMIC_TTNET 0.00000 0.5584 0.0000 1.000 0.86 (n/a) __T_BUG5868_RDNS_DYNAMIC_TTNET 0.00000 0.4841 0.0000 1.000 0.84 (n/a) __RDNS_DYNAMIC_DIALIN 0.00000 0.4839 0.0000 1.000 0.84 (n/a) __T_BUG5868_RDNS_DYNAMIC_DIALIN 0.00000 2.8357 0.1497 0.950 0.82 (n/a) __T_BUG5868_RDNS_STATIC 0.00000 38.3518 0.4023 0.990 0.79 (n/a) __RDNS_DYNAMIC_IPADDR 0.00000 38.3342 0.4046 0.990 0.79 (n/a) __T_BUG5868_RDNS_DYNAMIC_IPADDR 0.00000 0.1718 0.0000 1.000 0.78 (n/a) __RDNS_DYNAMIC_HEXIP 0.00000 0.1715 0.0000 1.000 0.78 (n/a) __T_BUG5868_RDNS_DYNAMIC_HEXIP 0.00000 0.1639 0.0000 1.000 0.77 (n/a) __RDNS_DYNAMIC_HOME_NL 0.00000 0.1638 0.0000 1.000 0.77 (n/a) __T_BUG5868_RDNS_DYNAMIC_HOME_NL 0.00000 0.1548 0.0000 1.000 0.77 (n/a) __RDNS_DYNAMIC_CHELLO_NL 0.00000 0.1547 0.0000 1.000 0.77 (n/a) __T_BUG5868_RDNS_DYNAMIC_CHELLO_NL 0.00000 0.0410 0.0000 1.000 0.66 (n/a) __RDNS_DYNAMIC_NTL 0.00000 0.0409 0.0000 1.000 0.66 (n/a) __T_BUG5868_RDNS_DYNAMIC_NTL 0.00000 37.6707 2.8371 0.930 0.63 (n/a) __RDNS_NONE 0.00000 37.6472 2.8371 0.930 0.63 (n/a) __T_BUG5868_RDNS_NONE 0.00000 0.0176 0.0058 0.751 0.59 (n/a) __RDNS_DYNAMIC_SPACELAN 0.00000 0.0176 0.0058 0.751 0.59 (n/a) __T_BUG5868_RDNS_DYNAMIC_SPACELAN 0.00000 0.0147 0.0000 1.000 0.58 (n/a) __T_BUG5868_RDNS_DYNAMIC_OOL 0.00000 0.0147 0.0000 1.000 0.58 (n/a) __RDNS_DYNAMIC_OOL 0.00000 0.0131 0.0000 1.000 0.58 (n/a) __RDNS_DYNAMIC_ROGERS 0.00000 0.0131 0.0000 1.000 0.58 (n/a) __T_BUG5868_RDNS_DYNAMIC_ROGERS 0.00000 0.0036 0.0000 1.000 0.52 (n/a) __RDNS_DYNAMIC_ASAHI 0.00000 99.9994 97.8015 0.506 0.51 (n/a) __T_BUG5868_LAST_EXTERNAL_RELAY_NO_AUTH 0.00000 0.0007 0.0000 1.000 0.50 (n/a) __T_BUG5868_RDNS_DYNAMIC_TDS 0.00000 0.0007 0.0000 1.000 0.50 (n/a) __RDNS_DYNAMIC_TDS 0.00000 0.0001 0.0000 1.000 0.50 (n/a) __RDNS_DYNAMIC_VELOX 0.00000 0.0001 0.0000 1.000 0.50 (n/a) __T_BUG5868_RDNS_DYNAMIC_VELOX 0.00000 0.0000 0.0000 0.500 0.50 (n/a) __RDNS_DYNAMIC_RR2 0.00000 0.0000 0.0000 0.500 0.50 (n/a) __RDNS_DYNAMIC_VTR 0.00000 0.0000 0.0000 0.500 0.50 (n/a) __RDNS_DYNAMIC_ADELPHIA 0.00000 0.0000 0.0000 0.500 0.50 (n/a) __T_BUG5868_RDNS_DYNAMIC_COMCAST 0.00000 0.0000 0.0000 0.500 0.50 (n/a) __RDNS_DYNAMIC_VIRTUA 0.00000 0.0000 0.0000 0.500 0.50 (n/a) __T_BUG5868_RDNS_DYNAMIC_CHELLO_NO 0.00000 0.0000 0.0000 0.500 0.50 (n/a) __T_BUG5868_RDNS_DYNAMIC_VTR 0.00000 0.0000 0.0000 0.500 0.50 (n/a) __T_BUG5868_RDNS_DYNAMIC_YAHOOBB 0.00000 0.0000 0.0000 0.500 0.50 (n/a) __T_BUG5868_RDNS_DYNAMIC_ADELPHIA 0.00000 0.0000 0.0000 0.500 0.50 (n/a) __RDNS_DYNAMIC_CHELLO_NO 0.00000 0.0000 0.0000 0.500 0.50 (n/a) __T_BUG5868_RDNS_DYNAMIC_TELIA 0.00000 0.0000 0.0000 0.500 0.50 (n/a) __T_BUG5868_RDNS_DYNAMIC_ATTBI 0.00000 0.0000 0.0000 0.500 0.50 (n/a) __RDNS_DYNAMIC_COMCAST 0.00000 0.0000 0.0000 0.500 0.50 (n/a) __RDNS_DYNAMIC_TELIA 0.00000 0.0000 0.0000 0.500 0.50 (n/a) __RDNS_DYNAMIC_ATTBI 0.00000 0.0000 0.0000 0.500 0.50 (n/a) __T_BUG5868_RDNS_DYNAMIC_VIRTUA 0.00000 0.0000 0.0000 0.500 0.50 (n/a) __RDNS_DYNAMIC_YAHOOBB 0.00000 0.0000 0.0000 0.500 0.50 (n/a) __T_BUG5868_RDNS_DYNAMIC_RR2
Created attachment 4310 [details] DNSEval.pm fixes Fixed a few issues with DNSEval.pm, any comments for the first one especially? - Trusted relays should not be checked in (non-nice?) RBLs. When -lastexternal was a trusted host, it was still checked. - num_check_received was not checked for -notfirsthop - Redundant check removed --> if (scalar @ips + scalar @originating > 0) {
*bump* Also, do you want a patch for the rules or are they handled already? I assume the change is accepted?
Daryl, what do you think? (In reply to comment #12) > Fixed a few issues with DNSEval.pm, any comments for the first one especially? > > - Trusted relays should not be checked in (non-nice?) RBLs. When -lastexternal > was a trusted host, it was still checked. makes sense to me. > - num_check_received was not checked for -notfirsthop > > - Redundant check removed --> if (scalar @ips + scalar @originating > 0) { I'll need to think about these when I get a chance...
(In reply to comment #14) > Daryl, what do you think? > > (In reply to comment #12) > > Fixed a few issues with DNSEval.pm, any comments for the first one especially? > > > > - Trusted relays should not be checked in (non-nice?) RBLs. When -lastexternal > > was a trusted host, it was still checked. > > makes sense to me. I'd like to see an example (headers & tests run) of the current behaviour and an explanation of why it should be changed. It's not clear to me why you wouldn't just change the rule to be -firsttrusted... probably because I'm not sure which rules are causing the apparent problem. (It's also not clear to me why I called it -lastexternal and not -firstexternal, but that's something else entirely).
(In reply to comment #15) > > I'd like to see an example (headers & tests run) of the current behaviour and > an explanation of why it should be changed. It's not clear to me why you > wouldn't just change the rule to be -firsttrusted... probably because I'm not > sure which rules are causing the apparent problem. host1 -> host2 -> host3 internal_networks host1 trusted_networks host1 host2 Say in this case host2 is a big isp server that I trust and don't want to query RBLs. Host3 is a dynamic user that I don't want to match on HELO checks etc, hence all the discussion and change to lastexternal. This is all fine. All 'rbl-lastexternal' rules refer to host2. But they don't care that out2 is trusted and DNS queries are done. This is the thing I fixed now.
I just had an idea that also all these changes would allow. It is simple to convert all dnswl.org data to trusted_networks entries. There would be quite a large number of unneeded queries dropped, imagine it on a global level. I don't know how fast the cidr code in SA currently is, but I guess it should handle 30k entries. I'll be testing for sure..
Ok it seems the cidr chokes on it. But I guess I'll test around and open a new feature request after these are applied.
Created attachment 4339 [details] Remove unneeded if-clause + whitespace (for SVN)
Created attachment 4340 [details] Check trusted relays only on nice bls, fix num_check_received (for SVN) Here are the DNSEval diffs against SVN. For clarity, I splitted up it in two.
Compare bug 5987
Created attachment 4397 [details] rdns checks from external Patch for 3.3 sa-update 20_dynrdns.cf. As already tested, this should be ok. Dynamic reverse dns rules should match lastexternal and nothing else.
Created attachment 4398 [details] rdns checks from external Doh, here is the patch with corrent documentation (trusted -> internal).
Created attachment 4399 [details] helo checks from external (20_fake_helo_tests.cf) For 3.3 sa-update
Created attachment 4400 [details] lastexternal fixes for active rules (72_active.cf) Some bad ones here too.
patches still waiting..
(In reply to comment #26) > patches still waiting.. ok, applied: : uid=jm Sun Mar 29 22:28:29 UTC 2009; cd /home/jm/ftp/spamassassin : 23...; svn commit -m "bug 5856: replace almost all use of 'trusted_networks' in rules with 'internal_networks'; thanks to Henrik Krohns" Sending lib/Mail/SpamAssassin/Plugin/DNSEval.pm Sending rules/20_dynrdns.cf Sending rules/20_fake_helo_tests.cf Transmitting file data ... Committed revision 759794. : uid=jm Sun Mar 29 22:30:36 UTC 2009; cd /home/jm/ftp/spamassassin : 24...; svn commit -m "bug 5856: replace almost all use of 'trusted_networks' in rules with 'internal_networks'; thanks to Henrik Krohns" rulesrc/ Sending rulesrc/sandbox/dos/70_other.cf Sending rulesrc/sandbox/emailed/00_FVGT_File001.cf Sending rulesrc/sandbox/jm/20_basic.cf Sending rulesrc/sandbox/kb/20_header.cf Transmitting file data .... Committed revision 759795. I'm still not 100% on the idea, but after a bit of thinking I decided it makes more sense than the previous, so it's an improvement. the ruleqa results are good at least.
hmm. just one more thing -- are there any now-incorrect lines in the documentation/wiki as a result of this?
Well, the main point was fixing the rules to match current documentation. ;) But atleast TrustedRelays page could use some improvement. I'm sure many will miss the last tiny paragraph mentioning external usage. I'm bad at writing sensible documentation, there should be more examples of what all is possible..
Btw, the rule changes aren't yet in 3.3 sa-update. Forgotten or waiting?
(In reply to comment #30) > Btw, the rule changes aren't yet in 3.3 sa-update. Forgotten or waiting? 3.3.0 updates aren't really being published yet, as it hasn't been released yet....
I'm confused.. there are no rule updates for CVS (3.3)? I could swear I've seen sa-update get new ones once in a while.