Bug 6269 - FH_DATE_PAST_20XX scores on all mails dated 2010 or later.
Summary: FH_DATE_PAST_20XX scores on all mails dated 2010 or later.
Status: RESOLVED FIXED
Alias: None
Product: Spamassassin
Classification: Unclassified
Component: Rules (show other bugs)
Version: 3.2.5
Hardware: PC Linux
: P1 critical
Target Milestone: 3.2.6
Assignee: SpamAssassin Developer Mailing List
URL:
Whiteboard:
Keywords:
: 6270 (view as bug list)
Depends on: 5852
Blocks:
  Show dependency tree
 
Reported: 2010-01-01 02:15 UTC by Per Jessen
Modified: 2010-06-24 08:51 UTC (History)
18 users (show)



Attachment Type Modified Status Actions Submitter/CLA Status
Please ignore. Attachment spam. text/plain None mehmet ali gokbas [NoCLA]

Note You need to log in before you can comment on or make changes to this bug.
Description Per Jessen 2010-01-01 02:15:07 UTC
Promptly at the start of the new year, all mails started getting an extra 3.4 points based on FH_DATE_PAST_20XX:

header   FH_DATE_PAST_20XX      Date =~ /20[1-9][0-9]/ [if-unset: 2006]
describe FH_DATE_PAST_20XX      The date is grossly in the future.

It doesn't make much sense to have hardcoded dates like this in the rule-set.
Comment 1 AXB 2010-01-01 04:54:51 UTC
I've commented out the rule in
/rulesrc/sandbox/emailed/00_FVGT_File001.cf

Could someone pls push a fix for SA 3.2.x?
Comment 2 Henrik Krohns 2010-01-01 05:15:57 UTC
There was no need to comment it out. It was already fixed 5 months ago. :-)

http://svn.apache.org/viewvc/spamassassin/trunk/rulesrc/sandbox/emailed/00_FVGT_File001.cf?r1=794319&r2=796216&diff_format=h
Comment 3 Per Jessen 2010-01-01 05:22:59 UTC
(In reply to comment #2)
> There was no need to comment it out. It was already fixed 5 months ago. :-)
> 
> http://svn.apache.org/viewvc/spamassassin/trunk/rulesrc/sandbox/emailed/00_FVGT_File001.cf?r1=794319&r2=796216&diff_format=h

Not really much of a "fix" - more like a work-around that'll come back and bite again in  10 years. "grossly in the future" is directly related to the current time, so shouldn't this rule take the current time into account?
Comment 4 Henrik Krohns 2010-01-01 05:27:17 UTC
Right. But we have years of time to fix it.

And it appears there is already a eval function for it: check_for_shifted_date
Comment 5 Per Jessen 2010-01-01 05:33:19 UTC
Okay, that's cool - any idea why the rule wasn't updated with sa-update?
Comment 6 Rob Janssen 2010-01-01 05:34:42 UTC
(In reply to comment #2)
> There was no need to comment it out. It was already fixed 5 months ago. :-)

Unfortunately this fix is neither in the released version of SpamAssassin, nor in the ruleset retrieved when sa-update is run.

This means that a lot of SpamAssassin installations in the world are running in trouble today.

I think this bug requires more attention than this.
Comment 7 Sebastian 2010-01-01 08:38:53 UTC
(In reply to comment #6)
> (In reply to comment #2)
> > There was no need to comment it out. It was already fixed 5 months ago. :-)
> 
> Unfortunately this fix is neither in the released version of SpamAssassin, nor
> in the ruleset retrieved when sa-update is run.
> 
> This means that a lot of SpamAssassin installations in the world are running in
> trouble today.

I agree, this needs to get fixed in the sa-update rulesets ASAP. It's causing a lot of false positives!
Comment 8 Klaus Steinberger 2010-01-01 11:02:24 UTC
Has somebody forgotten to update the channels since a longer time?

5.2.3.updates.spamassassin.org returns 795855, which was at least the same number since Aug 2009!
Comment 9 Justin Mason 2010-01-01 12:51:52 UTC
damn, we thoroughly dropped the ball on fixing this properly in bug 5852 :(  This needed to be pushed to 3.2 updates, and never was.


: 95...; svn commit -m "bug 6269: fix FH_DATE_PAST_20XX in 3.2 updates as well as trunk and 3.2 release branch"
Sending        rules-3.2/72_active.cf
Transmitting file data .
Committed revision 895073.
Comment 10 Warren Togami 2010-01-01 12:52:59 UTC
*** Bug 6270 has been marked as a duplicate of this bug. ***
Comment 11 Justin Mason 2010-01-01 12:54:15 UTC
3.2 rule update pushed.
Comment 12 Justin Mason 2010-01-01 13:01:42 UTC
this should now be fixed -- sa-update for any 3.2.x release will pick up the fixed version, as soon as the DNS caching propagates.
Comment 13 Casper 2010-01-01 13:26:35 UTC
(In reply to comment #12)
> this should now be fixed -- sa-update for any 3.2.x release will pick up the
> fixed version, as soon as the DNS caching propagates.

Not entirely it seems. The one mirror that I found to be alive does have the new revision file at http://www.sa-update.pccc.com/895063.tar.gz but the content of 72_active.cf is still wrong. 
header   FH_DATE_PAST_20XX      Date =~ /20[1-9][0-9]/ [if-unset: 2006]

Am I missing something or did the push to one or more mirrors go wrong?
Comment 14 Casper 2010-01-01 13:29:30 UTC
(In reply to comment #13)
> Am I missing something or did the push to one or more mirrors go wrong?

Nevermind, I'm indeed missing something. Different update/revision.
Comment 15 Justin Mason 2010-01-01 14:02:29 UTC
bug 6271 is for the more long-term fix; removing the rule.
Comment 16 Rob Janssen 2010-01-01 14:10:58 UTC
I ran sa-update, it retrieved the rules (exitstatus 0) and as far as I can see the problem is still present.  Rule is still /20[1-9][0-9]/ and scores as before.
What is happening?
Comment 17 Jonathan 2010-01-01 14:21:28 UTC
(In reply to comment #16)
> I ran sa-update, it retrieved the rules (exitstatus 0) and as far as I can see
> the problem is still present.  Rule is still /20[1-9][0-9]/ and scores as
> before.
> What is happening?

It works for me. Are you looking in the proper place?

What is the output of:

grep FH_DATE_PAST_20XX /var/lib/spamassassin/3.002005/72_active.cf
Comment 18 Rob Janssen 2010-01-01 14:30:15 UTC
(In reply to comment #17)

> grep FH_DATE_PAST_20XX /var/lib/spamassassin/3.002005/72_active.cf

On my system the path is: /var/lib/spamassassin/3.002005/updates_spamassassin_org/72_active.cf

After seeing it was wrong, I manually edited the file to fix it.
But as I said, it still had the old pattern.  But the filedate was new, i.e. it was updated.

Now it is:
##{ FH_DATE_PAST_20XX
header   FH_DATE_PAST_20XX      Date =~ /20[2-9][0-9]/ [if-unset: 2006]
describe FH_DATE_PAST_20XX      The date is grossly in the future.
##} FH_DATE_PAST_20XX

(I changed the 1-9 into 2-9)
Comment 19 Jonathan 2010-01-01 14:42:31 UTC
(In reply to comment #18)
> On my system the path is:
> /var/lib/spamassassin/3.002005/updates_spamassassin_org/72_active.cf

Oops, my bad.

> After seeing it was wrong, I manually edited the file to fix it.

That is not what I asked for. Next time please take care in following instructions as we need to diagnose when things go wrong in order to see if it is a bug or a failure on your side. What you just did was covering your tracks and destroying trails.

> But as I said, it still had the old pattern.  But the filedate was new, i.e. > it was updated.

Do you still have the log of the update? Does it show something like this:

[11358] dbg: channel: metadata version = 795855
[11358] dbg: dns: 5.2.3.updates.spamassassin.org => 895075, parsed as 895075

> Now it is:
> ##{ FH_DATE_PAST_20XX
> header   FH_DATE_PAST_20XX      Date =~ /20[2-9][0-9]/ [if-unset: 2006]
> describe FH_DATE_PAST_20XX      The date is grossly in the future.
> ##} FH_DATE_PAST_20XX
> 
> (I changed the 1-9 into 2-9)

This will not stick normally and should be reset to whatever the update channels holds on the next run of sa_udpate.
Comment 20 Rob Janssen 2010-01-01 14:53:01 UTC
(In reply to comment #19)

> > After seeing it was wrong, I manually edited the file to fix it.
> 
> That is not what I asked for. Next time please take care in following
> instructions as we need to diagnose when things go wrong in order to see if it
> is a bug or a failure on your side. What you just did was covering your tracks
> and destroying trails.

I need to keep my system operational.  So when I see the system is again dropping mails, I fix it.  I did that before I read your comment.

> Do you still have the log of the update? Does it show something like this:

The sa-update does not seem to log anything.  I checked /var/log/mail and /var/log/messages.

I see in the proxy log file that it has retrieved this file:
http://www.sa-update.pccc.com/895063.tar.gz

When I retrieve that file manually, it indeed contains the wrong rule.

> This will not stick normally and should be reset to whatever the update
> channels holds on the next run of sa_udpate.

I know that, but I hope that the update channel will be fixed so that I get the correct rules when I run sa-update again.
Comment 21 Justin Mason 2010-01-01 14:57:54 UTC
(In reply to comment #20)
> I know that, but I hope that the update channel will be fixed so that I get the
> correct rules when I run sa-update again.

wait and try in a few minutes.  895075 is the current version, but there's quite a lot of caching and retrying involved.
Comment 22 Nils Breunese 2010-01-01 15:37:36 UTC
The name FH_DATE_PAST_20XX isn't very accurate, is it? Shouldn't the rule be renamed to FH_DATE_PAST_2019 (for the new /20[2-9][0-9]/ regex)?
Comment 23 Tony Hoyle 2010-01-01 15:59:16 UTC
(In reply to comment #21)
> (In reply to comment #20)
> > I know that, but I hope that the update channel will be fixed so that I get the
> > correct rules when I run sa-update again.
> 
> wait and try in a few minutes.  895075 is the current version, but there's
> quite a lot of caching and retrying involved.

I just updated to that version and this bug is not fixed.

[10722] dbg: channel: metadata version = 759778
[10722] dbg: dns: 5.2.3.updates.spamassassin.org => 895075, parsed as 895075
..
[10722] dbg: extracting: /tmp/.spamassassin10722b0EYz5tmp/72_active.cf

The 72_active.cf in that file contains the old definition:

##{ FH_DATE_PAST_20XX
header   FH_DATE_PAST_20XX      Date =~ /20[1-9][0-9]/ [if-unset: 2006]
describe FH_DATE_PAST_20XX      The date is grossly in the future.
##} FH_DATE_PAST_20XX
Comment 24 Nils Breunese 2010-01-01 16:04:00 UTC
(In reply to comment #23)

I ran sa-update on our boxes, they got updated to 895075 and it seems they got the update alright:

----
# grep FH_DATE_PAST_20XX /var/lib/spamassassin/3.002005/updates_spamassassin_org/72_active.cf 
##{ FH_DATE_PAST_20XX
header   FH_DATE_PAST_20XX      Date =~ /20[2-9][0-9]/ [if-unset: 2006]
describe FH_DATE_PAST_20XX      The date is grossly in the future.
##} FH_DATE_PAST_20XX
----
Comment 25 Aiko Barz 2010-01-03 10:06:07 UTC
Why don't you use
> header   FH_DATE_PAST_20XX eval:check_for_shifted_date('8760', 'undef')
instead of
> header   FH_DATE_PAST_20XX  Date =~ /20[2-9][0-9]/ [if-unset: 2006]
?

The question was asked here:
http://www.heise.de/newsticker/foren/S-Das-Peinliche-daran-ist/forum-171865/msg-17874835/read/

Otherwise: See you in 10 years again. :)
Comment 26 Justin Mason 2010-01-03 14:10:16 UTC
(In reply to comment #25)
> Why don't you use
> > header   FH_DATE_PAST_20XX eval:check_for_shifted_date('8760', 'undef')
> instead of
> > header   FH_DATE_PAST_20XX  Date =~ /20[2-9][0-9]/ [if-unset: 2006]
> ?
> 
> The question was asked here:
> http://www.heise.de/newsticker/foren/S-Das-Peinliche-daran-ist/forum-171865/msg-17874835/read/
> 
> Otherwise: See you in 10 years again. :)

As has already been said, see bug 6272 for the long-term fix (ie. probable removal).  This tracks the short-term band-aid only.  Please read the comments next time.

BTW, if anyone else has run sa-update, restarted spamd/amavis, and is still seeing hits on the old rule -- check to ensure you don't have the 00_FVGT_File001.cf SARE rules file around.  That file also contains a copy of the rule.
Comment 27 tm 2010-01-04 08:35:45 UTC
What about the if-unset part? At first glance, it looks like this number should be bumped, too.
Comment 28 Justin Mason 2010-01-04 10:07:32 UTC
(In reply to comment #27)
> What about the if-unset part? At first glance, it looks like this number should
> be bumped, too.

nope, 2006 !~ /2020/
Comment 29 jbassett 2010-01-09 01:29:40 UTC
I have updated my rules by running sa-update and have confirmed that the new rules have been installed.

ls -lah gives:

Jan  9 09:01 /var/lib/spamassassin/3.002001/updates_spamassassin_org/72_active.cf

Editing the file shows the 1-9 has indeed been changed to 2-9.

I have even restarted spamd but my email logs show all email still has a FH_DATE_PAST_20XX value of 3.188 which is kicking some legitimate email out as spam.
Comment 30 Justin Mason 2010-01-09 14:14:20 UTC
(In reply to comment #29)
> I have updated my rules by running sa-update and have confirmed that the new
> rules have been installed.
> 
> ls -lah gives:
> 
> Jan  9 09:01
> /var/lib/spamassassin/3.002001/updates_spamassassin_org/72_active.cf
> 
> Editing the file shows the 1-9 has indeed been changed to 2-9.
> 
> I have even restarted spamd but my email logs show all email still has a
> FH_DATE_PAST_20XX value of 3.188 which is kicking some legitimate email out as
> spam.

have you checked to ensure you aren't using an old copy of the 00_FVGT_File001.cf SARE rules file?
Comment 31 jbassett 2010-01-09 14:38:24 UTC
After your suggestion I found a folder called saupdates_openprotect_com in /var/lib/spamassassin containing a range of files such as 70_sare_adult.cf.

I have now removed this folder and restarted spamd but the problem persists.

The files in that folder appear to be the only place where a reference to sare exists or is there a .cf file I should look in?  I was unable to find anything relevant in local.cf.
Comment 32 Sidney Markowitz 2010-01-10 03:07:43 UTC
(in reply to comment #31)

Are you running with compiled rules and need to run sa-compile after having updated the rules?

If not, try running spamd with the option -Dconfig,zoom

Tha will output "zoom" level messages to the log if compiled rules are being used, as well as output config  level messages that list every configuration file that is loaded so you can search all of them.
Comment 33 jbassett 2010-01-10 03:35:19 UTC
Running "spamd -Dconfig,zoom" outputs many lines including ones of this nature, then though I have removed the directory "saupdates_openprotect_com":

[4954] dbg: config: fixed relative path: /var/lib/spamassassin/3.002001/saupdates_openprotect_com/70_sare_adult.cf

I have also run sa-compile and only after doing so did a "compile" directory appear in /var.lib/spamassassin so I do not think I was using compiled rules at all.

I had a look through the logs but was unable to find anything related to "zoom".
Comment 34 jbassett 2010-01-10 03:40:15 UTC
Seems I overlooked a file that was referencing the sare rules, I have now removed it and no reference to the sare rules directory is now present when I run spamd.  

Logs still show all email getting FH_DATE_PAST_20XX=3.188 though.
Comment 35 jbassett 2010-01-10 03:52:52 UTC
I have tried commenting out the rogue rule to disable it entirely and running sa-compile.  Still get 3.188 on the score.

Also added "meta FH_DATE_PAST_20XX (0)" to 72_removed.cf and run sa-compile.  Still get the score.

That seems to indicate that it is not the sa-update or sa-compiled rules that are adding the 3.188 score?
Comment 36 Sidney Markowitz 2010-01-10 05:06:57 UTC
(in reply to comment #33 and comment #34 and comment #35)

If there are no "zoom" log messages even though zoom is in the -D option, then
you are not using compiled rules as you suspected. Running sa-compile now
didn't do anything negative or positive as you aren't using it, so that's fine.

So you don't have to think about sa-compile at all. It is not enabled in the v320.pre file and isn't a factor.

The "config" debug produces many log lines as you saw, but the important ones
for this purpose are the ones that say "dbg: config: read file" and a file
name. One of those files has to contain the offending old definition of
FH_DATE_PAST_20XX.

Either gather up all those file names, or see what directories that are in and
search *.cf files in those directories, but in either case grep those files for
FH_DATE_PAST_20XX as it has to be in there somewhere with the old definition.

Whatever file the rule definition is in, whether or not is is being brought in by sa-update, there has to be a corresponding "dbg: config: read file" log message for that file.
Comment 37 jbassett 2010-01-10 06:12:58 UTC
Seems to be working ok now.

I have been restarting spamd after each change, instead it seems I should have been restarting amavisd instead of or as well as spamd!

At least its a lesson learned, thank you for all your help and its a good heads up for me to keep an eye out for possible similar issues at the start of 2011 :-)

J
Comment 38 Mark Martinec 2010-01-10 08:31:35 UTC
> I have been restarting spamd after each change, instead it seems
> I should have been restarting amavisd instead of or as well as spamd!

It makes not sense to run spamd if you have amavisd running.
Use one or the other, not both. The difference between the two is mainly
in how they interface with MTA: amavisd speaks SMTP, spamd speaks
spamc/spamd protocol.
Comment 39 djb 2010-01-14 03:33:28 UTC
Hi,

Ran an sa-update to fix this issue and it completed without error.

However on checking my config files I find the following:-

#find . -name 72_active.cf
./root/duncan/Mail-SpamAssassin-3.2.5/rules/72_active.cf
./var/lib/spamassassin/3.002005/updates_spamassassin_org/72_active.cf
./usr/share/spamassassin/72_active.cf

grep FH_DATE_PAST_20XX ./var/lib/spamassassin/3.002005/updates_spamassassin_org/72_active.cf
##{ FH_DATE_PAST_20XX
header   FH_DATE_PAST_20XX      Date =~ /20[2-9][0-9]/ [if-unset: 2006]
describe FH_DATE_PAST_20XX      The date is grossly in the future.
##} FH_DATE_PAST_20XX

# grep FH_DATE_PAST_20XX ./usr/share/spamassassin/72_active.cf      ##{ FH_DATE_PAST_20XX
header   FH_DATE_PAST_20XX      Date =~ /20[1-9][0-9]/ [if-unset: 2006]
describe FH_DATE_PAST_20XX      The date is grossly in the future.
##} FH_DATE_PAST_20XX

As you can see the one in /usr/share has not been updated.

I have disabled this rule for now in my local.cf
Comment 40 Sidney Markowitz 2010-01-14 04:25:09 UTC
(reply to comment #39)

See the FAQ at http://wiki.apache.org/spamassassin/RuleUpdates especially the first FAQ there. If the rules update directory exists SpamAssassin does not read rules from the default rules directory.
Comment 41 Mark Martinec 2010-01-14 04:28:39 UTC
(In reply to comment #39)
> Ran an sa-update to fix this issue and it completed without error.
> However on checking my config files I find the following:-
> As you can see the one in /usr/share has not been updated.

Yes, this is normal, it's how it is supposed to work.
The rules in /usr/share/spamassassin are not consulted
if a directory /var/lib/spamassassin/3.x exists.
The sa-update only updates the later.

> I have disabled this rule for now in my local.cf

Can't hurt, although it was not necessary.
Comment 42 djb 2010-01-14 05:51:29 UTC
Thanks for the quick response and the info re sa-update - makes sense - sorry I did not RTFM (or is that RTFFAQ?).

Duncan
Comment 43 Justin Mason 2010-01-14 05:53:19 UTC
OK -- anyone who has more "I ran sa-update but the rule still exists" problems -- please mail the users@ mailing list first, rather than commenting on this bug.  thanks.
Comment 44 mehmet ali gokbas 2010-06-24 07:33:16 UTC
Created attachment 4785 [details]
Please ignore. Attachment spam.

test mail
Comment 45 Karsten Bräckelmann 2010-06-24 08:51:42 UTC
Re comment 44: Account suspended for attachment spamming. Bugmail Disabled. Renamed attachment and marked obsolete.

Getting desperate, eh? Attachments are not directly viewable in a browser.