SA Bugzilla – Bug 6269
FH_DATE_PAST_20XX scores on all mails dated 2010 or later.
Last modified: 2010-06-24 08:51:42 UTC
Promptly at the start of the new year, all mails started getting an extra 3.4 points based on FH_DATE_PAST_20XX: header FH_DATE_PAST_20XX Date =~ /20[1-9][0-9]/ [if-unset: 2006] describe FH_DATE_PAST_20XX The date is grossly in the future. It doesn't make much sense to have hardcoded dates like this in the rule-set.
I've commented out the rule in /rulesrc/sandbox/emailed/00_FVGT_File001.cf Could someone pls push a fix for SA 3.2.x?
There was no need to comment it out. It was already fixed 5 months ago. :-) http://svn.apache.org/viewvc/spamassassin/trunk/rulesrc/sandbox/emailed/00_FVGT_File001.cf?r1=794319&r2=796216&diff_format=h
(In reply to comment #2) > There was no need to comment it out. It was already fixed 5 months ago. :-) > > http://svn.apache.org/viewvc/spamassassin/trunk/rulesrc/sandbox/emailed/00_FVGT_File001.cf?r1=794319&r2=796216&diff_format=h Not really much of a "fix" - more like a work-around that'll come back and bite again in 10 years. "grossly in the future" is directly related to the current time, so shouldn't this rule take the current time into account?
Right. But we have years of time to fix it. And it appears there is already a eval function for it: check_for_shifted_date
Okay, that's cool - any idea why the rule wasn't updated with sa-update?
(In reply to comment #2) > There was no need to comment it out. It was already fixed 5 months ago. :-) Unfortunately this fix is neither in the released version of SpamAssassin, nor in the ruleset retrieved when sa-update is run. This means that a lot of SpamAssassin installations in the world are running in trouble today. I think this bug requires more attention than this.
(In reply to comment #6) > (In reply to comment #2) > > There was no need to comment it out. It was already fixed 5 months ago. :-) > > Unfortunately this fix is neither in the released version of SpamAssassin, nor > in the ruleset retrieved when sa-update is run. > > This means that a lot of SpamAssassin installations in the world are running in > trouble today. I agree, this needs to get fixed in the sa-update rulesets ASAP. It's causing a lot of false positives!
Has somebody forgotten to update the channels since a longer time? 5.2.3.updates.spamassassin.org returns 795855, which was at least the same number since Aug 2009!
damn, we thoroughly dropped the ball on fixing this properly in bug 5852 :( This needed to be pushed to 3.2 updates, and never was. : 95...; svn commit -m "bug 6269: fix FH_DATE_PAST_20XX in 3.2 updates as well as trunk and 3.2 release branch" Sending rules-3.2/72_active.cf Transmitting file data . Committed revision 895073.
*** Bug 6270 has been marked as a duplicate of this bug. ***
3.2 rule update pushed.
this should now be fixed -- sa-update for any 3.2.x release will pick up the fixed version, as soon as the DNS caching propagates.
(In reply to comment #12) > this should now be fixed -- sa-update for any 3.2.x release will pick up the > fixed version, as soon as the DNS caching propagates. Not entirely it seems. The one mirror that I found to be alive does have the new revision file at http://www.sa-update.pccc.com/895063.tar.gz but the content of 72_active.cf is still wrong. header FH_DATE_PAST_20XX Date =~ /20[1-9][0-9]/ [if-unset: 2006] Am I missing something or did the push to one or more mirrors go wrong?
(In reply to comment #13) > Am I missing something or did the push to one or more mirrors go wrong? Nevermind, I'm indeed missing something. Different update/revision.
bug 6271 is for the more long-term fix; removing the rule.
I ran sa-update, it retrieved the rules (exitstatus 0) and as far as I can see the problem is still present. Rule is still /20[1-9][0-9]/ and scores as before. What is happening?
(In reply to comment #16) > I ran sa-update, it retrieved the rules (exitstatus 0) and as far as I can see > the problem is still present. Rule is still /20[1-9][0-9]/ and scores as > before. > What is happening? It works for me. Are you looking in the proper place? What is the output of: grep FH_DATE_PAST_20XX /var/lib/spamassassin/3.002005/72_active.cf
(In reply to comment #17) > grep FH_DATE_PAST_20XX /var/lib/spamassassin/3.002005/72_active.cf On my system the path is: /var/lib/spamassassin/3.002005/updates_spamassassin_org/72_active.cf After seeing it was wrong, I manually edited the file to fix it. But as I said, it still had the old pattern. But the filedate was new, i.e. it was updated. Now it is: ##{ FH_DATE_PAST_20XX header FH_DATE_PAST_20XX Date =~ /20[2-9][0-9]/ [if-unset: 2006] describe FH_DATE_PAST_20XX The date is grossly in the future. ##} FH_DATE_PAST_20XX (I changed the 1-9 into 2-9)
(In reply to comment #18) > On my system the path is: > /var/lib/spamassassin/3.002005/updates_spamassassin_org/72_active.cf Oops, my bad. > After seeing it was wrong, I manually edited the file to fix it. That is not what I asked for. Next time please take care in following instructions as we need to diagnose when things go wrong in order to see if it is a bug or a failure on your side. What you just did was covering your tracks and destroying trails. > But as I said, it still had the old pattern. But the filedate was new, i.e. > it was updated. Do you still have the log of the update? Does it show something like this: [11358] dbg: channel: metadata version = 795855 [11358] dbg: dns: 5.2.3.updates.spamassassin.org => 895075, parsed as 895075 > Now it is: > ##{ FH_DATE_PAST_20XX > header FH_DATE_PAST_20XX Date =~ /20[2-9][0-9]/ [if-unset: 2006] > describe FH_DATE_PAST_20XX The date is grossly in the future. > ##} FH_DATE_PAST_20XX > > (I changed the 1-9 into 2-9) This will not stick normally and should be reset to whatever the update channels holds on the next run of sa_udpate.
(In reply to comment #19) > > After seeing it was wrong, I manually edited the file to fix it. > > That is not what I asked for. Next time please take care in following > instructions as we need to diagnose when things go wrong in order to see if it > is a bug or a failure on your side. What you just did was covering your tracks > and destroying trails. I need to keep my system operational. So when I see the system is again dropping mails, I fix it. I did that before I read your comment. > Do you still have the log of the update? Does it show something like this: The sa-update does not seem to log anything. I checked /var/log/mail and /var/log/messages. I see in the proxy log file that it has retrieved this file: http://www.sa-update.pccc.com/895063.tar.gz When I retrieve that file manually, it indeed contains the wrong rule. > This will not stick normally and should be reset to whatever the update > channels holds on the next run of sa_udpate. I know that, but I hope that the update channel will be fixed so that I get the correct rules when I run sa-update again.
(In reply to comment #20) > I know that, but I hope that the update channel will be fixed so that I get the > correct rules when I run sa-update again. wait and try in a few minutes. 895075 is the current version, but there's quite a lot of caching and retrying involved.
The name FH_DATE_PAST_20XX isn't very accurate, is it? Shouldn't the rule be renamed to FH_DATE_PAST_2019 (for the new /20[2-9][0-9]/ regex)?
(In reply to comment #21) > (In reply to comment #20) > > I know that, but I hope that the update channel will be fixed so that I get the > > correct rules when I run sa-update again. > > wait and try in a few minutes. 895075 is the current version, but there's > quite a lot of caching and retrying involved. I just updated to that version and this bug is not fixed. [10722] dbg: channel: metadata version = 759778 [10722] dbg: dns: 5.2.3.updates.spamassassin.org => 895075, parsed as 895075 .. [10722] dbg: extracting: /tmp/.spamassassin10722b0EYz5tmp/72_active.cf The 72_active.cf in that file contains the old definition: ##{ FH_DATE_PAST_20XX header FH_DATE_PAST_20XX Date =~ /20[1-9][0-9]/ [if-unset: 2006] describe FH_DATE_PAST_20XX The date is grossly in the future. ##} FH_DATE_PAST_20XX
(In reply to comment #23) I ran sa-update on our boxes, they got updated to 895075 and it seems they got the update alright: ---- # grep FH_DATE_PAST_20XX /var/lib/spamassassin/3.002005/updates_spamassassin_org/72_active.cf ##{ FH_DATE_PAST_20XX header FH_DATE_PAST_20XX Date =~ /20[2-9][0-9]/ [if-unset: 2006] describe FH_DATE_PAST_20XX The date is grossly in the future. ##} FH_DATE_PAST_20XX ----
Why don't you use > header FH_DATE_PAST_20XX eval:check_for_shifted_date('8760', 'undef') instead of > header FH_DATE_PAST_20XX Date =~ /20[2-9][0-9]/ [if-unset: 2006] ? The question was asked here: http://www.heise.de/newsticker/foren/S-Das-Peinliche-daran-ist/forum-171865/msg-17874835/read/ Otherwise: See you in 10 years again. :)
(In reply to comment #25) > Why don't you use > > header FH_DATE_PAST_20XX eval:check_for_shifted_date('8760', 'undef') > instead of > > header FH_DATE_PAST_20XX Date =~ /20[2-9][0-9]/ [if-unset: 2006] > ? > > The question was asked here: > http://www.heise.de/newsticker/foren/S-Das-Peinliche-daran-ist/forum-171865/msg-17874835/read/ > > Otherwise: See you in 10 years again. :) As has already been said, see bug 6272 for the long-term fix (ie. probable removal). This tracks the short-term band-aid only. Please read the comments next time. BTW, if anyone else has run sa-update, restarted spamd/amavis, and is still seeing hits on the old rule -- check to ensure you don't have the 00_FVGT_File001.cf SARE rules file around. That file also contains a copy of the rule.
What about the if-unset part? At first glance, it looks like this number should be bumped, too.
(In reply to comment #27) > What about the if-unset part? At first glance, it looks like this number should > be bumped, too. nope, 2006 !~ /2020/
I have updated my rules by running sa-update and have confirmed that the new rules have been installed. ls -lah gives: Jan 9 09:01 /var/lib/spamassassin/3.002001/updates_spamassassin_org/72_active.cf Editing the file shows the 1-9 has indeed been changed to 2-9. I have even restarted spamd but my email logs show all email still has a FH_DATE_PAST_20XX value of 3.188 which is kicking some legitimate email out as spam.
(In reply to comment #29) > I have updated my rules by running sa-update and have confirmed that the new > rules have been installed. > > ls -lah gives: > > Jan 9 09:01 > /var/lib/spamassassin/3.002001/updates_spamassassin_org/72_active.cf > > Editing the file shows the 1-9 has indeed been changed to 2-9. > > I have even restarted spamd but my email logs show all email still has a > FH_DATE_PAST_20XX value of 3.188 which is kicking some legitimate email out as > spam. have you checked to ensure you aren't using an old copy of the 00_FVGT_File001.cf SARE rules file?
After your suggestion I found a folder called saupdates_openprotect_com in /var/lib/spamassassin containing a range of files such as 70_sare_adult.cf. I have now removed this folder and restarted spamd but the problem persists. The files in that folder appear to be the only place where a reference to sare exists or is there a .cf file I should look in? I was unable to find anything relevant in local.cf.
(in reply to comment #31) Are you running with compiled rules and need to run sa-compile after having updated the rules? If not, try running spamd with the option -Dconfig,zoom Tha will output "zoom" level messages to the log if compiled rules are being used, as well as output config level messages that list every configuration file that is loaded so you can search all of them.
Running "spamd -Dconfig,zoom" outputs many lines including ones of this nature, then though I have removed the directory "saupdates_openprotect_com": [4954] dbg: config: fixed relative path: /var/lib/spamassassin/3.002001/saupdates_openprotect_com/70_sare_adult.cf I have also run sa-compile and only after doing so did a "compile" directory appear in /var.lib/spamassassin so I do not think I was using compiled rules at all. I had a look through the logs but was unable to find anything related to "zoom".
Seems I overlooked a file that was referencing the sare rules, I have now removed it and no reference to the sare rules directory is now present when I run spamd. Logs still show all email getting FH_DATE_PAST_20XX=3.188 though.
I have tried commenting out the rogue rule to disable it entirely and running sa-compile. Still get 3.188 on the score. Also added "meta FH_DATE_PAST_20XX (0)" to 72_removed.cf and run sa-compile. Still get the score. That seems to indicate that it is not the sa-update or sa-compiled rules that are adding the 3.188 score?
(in reply to comment #33 and comment #34 and comment #35) If there are no "zoom" log messages even though zoom is in the -D option, then you are not using compiled rules as you suspected. Running sa-compile now didn't do anything negative or positive as you aren't using it, so that's fine. So you don't have to think about sa-compile at all. It is not enabled in the v320.pre file and isn't a factor. The "config" debug produces many log lines as you saw, but the important ones for this purpose are the ones that say "dbg: config: read file" and a file name. One of those files has to contain the offending old definition of FH_DATE_PAST_20XX. Either gather up all those file names, or see what directories that are in and search *.cf files in those directories, but in either case grep those files for FH_DATE_PAST_20XX as it has to be in there somewhere with the old definition. Whatever file the rule definition is in, whether or not is is being brought in by sa-update, there has to be a corresponding "dbg: config: read file" log message for that file.
Seems to be working ok now. I have been restarting spamd after each change, instead it seems I should have been restarting amavisd instead of or as well as spamd! At least its a lesson learned, thank you for all your help and its a good heads up for me to keep an eye out for possible similar issues at the start of 2011 :-) J
> I have been restarting spamd after each change, instead it seems > I should have been restarting amavisd instead of or as well as spamd! It makes not sense to run spamd if you have amavisd running. Use one or the other, not both. The difference between the two is mainly in how they interface with MTA: amavisd speaks SMTP, spamd speaks spamc/spamd protocol.
Hi, Ran an sa-update to fix this issue and it completed without error. However on checking my config files I find the following:- #find . -name 72_active.cf ./root/duncan/Mail-SpamAssassin-3.2.5/rules/72_active.cf ./var/lib/spamassassin/3.002005/updates_spamassassin_org/72_active.cf ./usr/share/spamassassin/72_active.cf grep FH_DATE_PAST_20XX ./var/lib/spamassassin/3.002005/updates_spamassassin_org/72_active.cf ##{ FH_DATE_PAST_20XX header FH_DATE_PAST_20XX Date =~ /20[2-9][0-9]/ [if-unset: 2006] describe FH_DATE_PAST_20XX The date is grossly in the future. ##} FH_DATE_PAST_20XX # grep FH_DATE_PAST_20XX ./usr/share/spamassassin/72_active.cf ##{ FH_DATE_PAST_20XX header FH_DATE_PAST_20XX Date =~ /20[1-9][0-9]/ [if-unset: 2006] describe FH_DATE_PAST_20XX The date is grossly in the future. ##} FH_DATE_PAST_20XX As you can see the one in /usr/share has not been updated. I have disabled this rule for now in my local.cf
(reply to comment #39) See the FAQ at http://wiki.apache.org/spamassassin/RuleUpdates especially the first FAQ there. If the rules update directory exists SpamAssassin does not read rules from the default rules directory.
(In reply to comment #39) > Ran an sa-update to fix this issue and it completed without error. > However on checking my config files I find the following:- > As you can see the one in /usr/share has not been updated. Yes, this is normal, it's how it is supposed to work. The rules in /usr/share/spamassassin are not consulted if a directory /var/lib/spamassassin/3.x exists. The sa-update only updates the later. > I have disabled this rule for now in my local.cf Can't hurt, although it was not necessary.
Thanks for the quick response and the info re sa-update - makes sense - sorry I did not RTFM (or is that RTFFAQ?). Duncan
OK -- anyone who has more "I ran sa-update but the rule still exists" problems -- please mail the users@ mailing list first, rather than commenting on this bug. thanks.
Created attachment 4785 [details] Please ignore. Attachment spam. test mail
Re comment 44: Account suspended for attachment spamming. Bugmail Disabled. Renamed attachment and marked obsolete. Getting desperate, eh? Attachments are not directly viewable in a browser.