Bug 6363 - URIBL_DBL added to UPDATE version 917919 for SA 3.3.0
Summary: URIBL_DBL added to UPDATE version 917919 for SA 3.3.0
Status: RESOLVED FIXED
Alias: None
Product: Spamassassin
Classification: Unclassified
Component: Rules (show other bugs)
Version: 3.3.0
Hardware: All All
: P1 critical
Target Milestone: 3.3.1
Assignee: SpamAssassin Developer Mailing List
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2010-03-03 17:37 UTC by Paul Fisher
Modified: 2010-03-12 14:11 UTC (History)
1 user (show)



Attachment Type Modified Status Actions Submitter/CLA Status

Note You need to log in before you can comment on or make changes to this bug.
Description Paul Fisher 2010-03-03 17:37:26 UTC
URIBL_DBL requires SpamAssassin 3.3.1 (see bug 6335); however, sa-update for 3.3.0 has pulled in URIBL_DBL, without a defined score.  All standard 3.3.0 installs with sa-update enabled now have spam scores that are inflated by 1 when SpamAssassin finds any URLs in a message.  SA 3.3.0 sends IP addresses to the DBL, which returns 127.0.1.255 (IP queries prohibited!), which causes URIBL_DBL to fire.

URIBL_DBL should not have been published for 3.3.0, and additionally, to safeguard against any future situations where SA receives 127.0.1.255 from the DBL, URIBL_DBL should never fire if 127.0.1.255 is returned from the DBL.
Comment 1 Justin Mason 2010-03-03 17:53:23 UTC
damn, I forgot that trunk rules are being promoted to 3.3.x!  have a fix ready...
Comment 2 Justin Mason 2010-03-03 17:54:20 UTC
Index: lib/Mail/SpamAssassin/Plugin/URIDNSBL.pm
===================================================================
--- lib/Mail/SpamAssassin/Plugin/URIDNSBL.pm    (revision 918599)
+++ lib/Mail/SpamAssassin/Plugin/URIDNSBL.pm    (working copy)
@@ -1002,4 +1002,8 @@
 
 # ---------------------------------------------------------------------------
 
+# capability checks for "if can()":
+#
+sub has_tflags_domains_only { 1 }
+
 1;
Index: rules/25_uribl.cf
===================================================================
--- rules/25_uribl.cf   (revision 918599)
+++ rules/25_uribl.cf   (working copy)
@@ -41,10 +41,12 @@
 
 # DBL, http://www.spamhaus.org/dbl/ .  Note that hits return 127.0.1.x
 # A records, so we use a 32-bit mask to match that /24 range.
+if can(Mail::SpamAssassin::Plugin::URIDNSBL::has_tflags_domains_only)
 urirhssub       URIBL_DBL       dbl.spamhaus.org.       A   2130706688
 body            URIBL_DBL       eval:check_uridnsbl('URIBL_DBL')
 describe        URIBL_DBL       Contains an URL listed in the DBL blocklist
 tflags          URIBL_DBL       net domains_only
+endif
 
 ###########################################################################
 ## SURBL


: 61...; svn commit -m "bug 6363: use 'if can()' to avoid publishing URIBL_DBL to 3.3.x updates prematurely" lib/Mail/SpamAssassin/Plugin/URIDNSBL.pm rules/25_uribl.cf
Sending        lib/Mail/SpamAssassin/Plugin/URIDNSBL.pm
Sending        rules/25_uribl.cf
Transmitting file data ..
Committed revision 918602.
Comment 3 Justin Mason 2010-03-03 17:59:54 UTC
building an update now.
Comment 4 Justin Mason 2010-03-03 18:04:08 UTC
update 918603 should NOT contain a version of URIBL_DBL that 3.3.x will attempt to use -- it's now protected with an "if can()" scope.

please let me know if you still see issues with that update (once it gets through the mirrors/DNS TTLs etc.)
Comment 5 Karsten Bräckelmann 2010-03-03 22:10:12 UTC
Also see bug 6335 comment 59.  (This bug is now mentioned on users@.)
Comment 6 Justin Mason 2010-03-04 00:30:29 UTC
ah.  it looks like there's some new update-generation code on updatesd on the zone: /home/updatesd/svn/mkupdates-with-scores/mkupdate-with-scores
so the update tarball I generated is useless. :(

Running that now...
Comment 7 Justin Mason 2010-03-04 00:58:05 UTC
"useless" as in generating for 3.4.0 only. 3.3.x updates were unaffected :(
Comment 8 Justin Mason 2010-03-04 10:25:59 UTC
I'm not sure that update-push worked.  Daryl, are you around to help debug this?
Comment 9 Justin Mason 2010-03-04 11:29:58 UTC
It appears to be breaking somewhere around here (pasting from cron mail):

+ STATUS=0
+ set -e
+ cd ..
+ rm -rf release_3.3.1 /tmp/sa-mkupdate-7499/release_3.3.1
+ return 0
+ update_dns_record 3.3.1 918364
+ SA_VERSION=3.3.1
+ UPDATE_REVISION=918364
++ echo 3.3.1
++ perl -pe 's/^(\d+)\.(\d+)\.(\d+)$/$3.$2.$1/'
+ RVERS=1.3.3
+ DNS_RECORD='1.3.3  TXT  "918364"'
+ echo 'DNS Record: 1.3.3  TXT  "918364"'
DNS Record: 1.3.3  TXT  "918364"
+ DNSFILE=/var/named/updates.spamassassin.org.d/3.3.1
+ mkdir /tmp/sa-mkupdate-7499/dns-backup
mkdir: Failed to make directory "/tmp/sa-mkupdate-7499/dns-backup"; File exists
+ set +e
+ cp /var/named/updates.spamassassin.org.d/3.3.1 /tmp/sa-mkupdate-7499/dns-backup/.
+ set -e
+ echo 1.3.3 TXT '"918364"'
+ mv /var/named/updates.spamassassin.org.d/3.3.1.mkupdate-with-scores.new /var/named/updates.spamassassin.org.d/3.3.1
+ return 0
+ UPDATED_VERSIONS=2
+ echo 'VERSIONS UPDATE PASSED ON: 2'
VERSIONS UPDATE PASSED ON: 2
+ '[' 2 -gt 0 ']'
+ EXIT=0
+ copy_update_paranoid /tmp/sa-mkupdate-7499/918364.tar.gz /var/www/buildbot.spamassassin.org/updatestage/918364.tar.gz
+ SRC=/tmp/sa-mkupdate-7499/918364.tar.gz
+ DST=/var/www/buildbot.spamassassin.org/updatestage/918364.tar.gz
+ set +e
+ cp /tmp/sa-mkupdate-7499/918364.tar.gz /var/www/buildbot.spamassassin.org/updatestage/918364.tar.gz
cp: cannot create /var/www/buildbot.spamassassin.org/updatestage/918364.tar.gz: Permission denied
+ diff -u /tmp/sa-mkupdate-7499/918364.tar.gz /var/www/buildbot.spamassassin.org/updatestage/918364.tar.gz
Binary files /tmp/sa-mkupdate-7499/918364.tar.gz and /var/www/buildbot.spamassassin.org/updatestage/918364.tar.gz differ
+ '[' 1 -ne 0 ']'
+ set -e
+ return 1
+ EXIT=5
+ '[' 5 -gt 0 ']'
+ (( I=0 ))
+ (( I<=1 ))
+ revert_dns_record 3.3.0
+ SA_VERSION=3.3.0
+ DNSFILE=/var/named/updates.spamassassin.org.d/3.3.0
+ set +e
+ cp /tmp/sa-mkupdate-7499/dns-backup/3.3.0 /var/named/updates.spamassassin.org.d/3.3.0
+ set -e
+ (( I++  ))
+ (( I<=1 ))
+ revert_dns_record 3.3.1
+ SA_VERSION=3.3.1
+ DNSFILE=/var/named/updates.spamassassin.org.d/3.3.1
+ set +e
+ cp /tmp/sa-mkupdate-7499/dns-backup/3.3.1 /var/named/updates.spamassassin.org.d/3.3.1
+ set -e
+ (( I++  ))
+ (( I<=1 ))
+ exit 5

also, 918364 is too old anyway.  the fix is in r918602.
Comment 10 Justin Mason 2010-03-04 11:31:45 UTC
no, wait, that's irrelevant -- it produced a working update for 3.3.0, just using the wrong rev:


Mar  4 03:28:09.212 [12658] dbg: diag: updates complete, exiting with code 0
+ STATUS=0
+ set -e
+ cd ..
+ rm -rf release_3.3.0 /tmp/sa-mkupdate-7499/release_3.3.0
+ return 0
+ update_dns_record 3.3.0 918364
+ SA_VERSION=3.3.0
+ UPDATE_REVISION=918364
++ echo 3.3.0
++ perl -pe 's/^(\d+)\.(\d+)\.(\d+)$/$3.$2.$1/'
+ RVERS=0.3.3
+ DNS_RECORD='0.3.3  TXT  "918364"'
+ echo 'DNS Record: 0.3.3  TXT  "918364"'
DNS Record: 0.3.3  TXT  "918364"
+ DNSFILE=/var/named/updates.spamassassin.org.d/3.3.0
+ mkdir /tmp/sa-mkupdate-7499/dns-backup
+ set +e
+ cp /var/named/updates.spamassassin.org.d/3.3.0 /tmp/sa-mkupdate-7499/dns-backup/.
+ set -e
+ echo 0.3.3 TXT '"918364"'
+ mv /var/named/updates.spamassassin.org.d/3.3.0.mkupdate-with-scores.new /var/named/updates.spamassassin.org.d/3.3.0
+ return 0
Comment 11 Justin Mason 2010-03-04 11:37:35 UTC
ok, it appears it takes the revision to check out directly from the mass-check results.  So there's no easy way currently to make an emergency rule change and cut a new update.  That's pretty vital. :(
Comment 12 Justin Mason 2010-03-04 11:57:15 UTC
kludging an emergency fix.  I copied the /home/updatesd/svn/spamassassin/build/mkupdates/run_part2 script, changed the version to 3.3.0, and ran it to cut a traditional-style update. 

It appears to have worked, and scp'ing the built files to another host and sa-update --install'ing them works fine, but would appreciate additional validation once it hits the mirrors and starts going live.

I've commented out the /export/home/updatesd/svn/mkupdates-with-scores script invocations on the zone until we can be sure that the update cut will not contain the buggy rule, which I guess will depend on the SVN rev it tries to use.

Again, we _need_ urgently to have a way to cut an update package from current SVN, without having to go through the full mass-check/score-generation daily cycle. 24 hour turnaround on a rule fix is insufficient. :(   opened bug 6365 for that.
Comment 13 Mark Martinec 2010-03-11 19:35:10 UTC
Do I understand correctly that this is now either fixed or pushed off
to other problem reports:

- sa-update for 3.3.0 has pulled in URIBL_DBL, without a defined score;
- URIBL_DBL should not have been published for 3.3.0
  -> fixed by a change in Comment 2, Comment 4

- scores that are inflated by 1 when SpamAssassin finds any URLs in a message.
  SA 3.3.0 sends IP addresses to the DBL, which returns 127.0.1.255 (IP queries
  prohibited!), which causes URIBL_DBL to fire.
  -> fixed in Bug 6335

- to safeguard against any future situations where SA receives 127.0.1.255 from
  the DBL, URIBL_DBL should never fire if 127.0.1.255 is returned from the DBL
  -> fixed by Bug 6335 and Bug 6362, added a new rule URIBL_DBL_ERROR

- Again, we _need_ urgently to have a way to cut an update package from current
  SVN, without having to go through the full mass-check/score-generation daily
  cycle. 24 hour turnaround on a rule fix is insufficient.
  -> pushed off to Bug 6365 and Bug 6368	

and can be closed?
Comment 14 Justin Mason 2010-03-12 14:11:07 UTC
(In reply to comment #13)
> Do I understand correctly that this is now either fixed or pushed off
> to other problem reports:

yep!  closing