6838 – 404 error from sa-update since 19/09/2012

Bug 6838 - 404 error from sa-update since 19/09/2012

Summary: 404 error from sa-update since 19/09/2012

Status:	RESOLVED FIXED

Alias:	None

Product:	Spamassassin
Classification:	Unclassified
Component:	sa-update (show other bugs)
Version:	unspecified
Hardware:	PC Linux

Importance:	P2 major
Target Milestone:	Undefined
Assignee:	SpamAssassin Developer Mailing List

URL:
Whiteboard:
Keywords:

Duplicates (2):	6761 6906 (view as bug list)
Depends on:
Blocks:

Reported:	2012-09-23 06:38 UTC by patrice
Modified:	2013-07-08 19:35 UTC (History)
CC List:	9 users (show)

Attachment	Type	Modified	Status	Actions	Submitter/CLA Status
Warn only with the verbose option	patch			None	Jarkko
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.

Description patrice 2012-09-23 06:38:49 UTC

Since 19/09/2012, I received the error message every morning :

/etc/cron.daily/spamassassin:
http: GET http://daryl.dostech.ca/sa-update/asf/1387055.tar.gz request failed: 404 Not Found: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /sa-update/asf/1387055.tar.gz was not found on this server.</p> <hr> <address>Apache/2.2.6 (Fedora) Server at daryl.dostech.ca Port 80</address> </body></html>

It seems this mirror is not up-to-date.
A rapid search on Google confirms that the problem concerns many systems.

Comment 1 Martin von Wittich 2012-09-24 18:36:51 UTC

This problem affects all of our Debian servers, so I've taken the time to debug this issue. The root cause seems to be the following file: 

http://spamassassin.apache.org/updates/MIRRORED.BY
# test mirror: zone, cached via Coral
#http://buildbot.spamassassin.org.nyud.net:8090/updatestage/
http://daryl.dostech.ca/sa-update/asf/ weight=5
http://www.sa-update.pccc.com/ weight=5
http://sa-update.secnap.net/ weight=5

It lists all available mirrors that sa-update can use; the first listed mirror daryl.dostech.ca is broken. It still responds to HTTP requests, but it doesn't contain the most recent SA updates, while the other two mirrors do:

iserv ~ # HEAD http://daryl.dostech.ca/sa-update/asf/1388977.tar.gz | head -n1
404 Not Found
iserv ~ # HEAD http://www.sa-update.pccc.com/1388977.tar.gz | head -n1 
200 OK
iserv ~ # HEAD http://sa-update.secnap.net/1388977.tar.gz | head -n1
200 OK

I'd propose that the broken mirror is removed from this file for the time being, until someone has figured out why they aren't up to date any longer.

Comment 2 Darxus 2012-09-24 18:49:21 UTC

What problem does this cause?  Do you still get an update from a functioning mirror?

Comment 3 Martin von Wittich 2012-09-24 19:00:15 UTC

> What problem does this cause?

Whenever sa-update decides to use daryl.dostech.ca, the following error message is reported by cron:

From root@dev2.iserv.eu Mon Sep 24 06:44:26 2012
Return-path: <root@dev2.iserv.eu>
Envelope-to: root@dev2.iserv.eu
Delivery-date: Mon, 24 Sep 2012 06:44:26 +0200
Received: from root by iserv.dev2.iserv.eu with local (Exim 4.72)
        (envelope-from <root@dev2.iserv.eu>)
        id 1TG0Ww-0000fb-CE
        for root@dev2.iserv.eu; Mon, 24 Sep 2012 06:44:26 +0200
Date: Mon, 24 Sep 2012 06:44:26 +0200
Message-Id: <E1TG0Ww-0000fb-CE@iserv.dev2.iserv.eu>
From: root@dev2.iserv.eu (Cron Daemon)
To: root@dev2.iserv.eu
Subject: Cron <root@iserv> test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
Content-Type: text/plain; charset=UTF-8
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin>
X-Cron-Env: <HOME=/root>
X-Cron-Env: <LOGNAME=root>

/etc/cron.daily/spamassassin:
http: GET http://daryl.dostech.ca/sa-update/asf/1388977.tar.gz request failed: 404 Not Found: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /sa-update/asf/1388977.tar.gz was not found on this server.</p> <hr> <address>Apache/2.2.6 (Fedora) Server at daryl.dostech.ca Port 80</address> </body></html> 

SpamAssassin still updates though whenever sa-update decides to use one of the two other mirrors. On our development server it looks like this:

2012-09-18: 404 error
2012-09-19: 404 error
2012-09-20: 404 error
2012-09-21: updated successfully
2012-09-22: 404 error
2012-09-23: updated successfully
2012-09-24: 404 error

Comment 4 Kevin A. McGrail 2012-09-25 12:55:13 UTC

Does the same sa-update instance through a 404 error and then go onto the next mirror or are you saying you have to run it again?

What version of SA are you using?

This sounds like a dupe of 6761 and in short, sa-update is designed to have some mirrors out of sync and it should try other servers.

Regards,
KAM

Comment 5 Martin von Wittich 2012-09-25 13:13:48 UTC

> What version of SA are you using?

We're using SpamAssassin 3.3.1, as provided by Debian stable:

iserv ~ # LANG=C apt-cache policy spamassassin
spamassassin:
  Installed: 3.3.1-1
  Candidate: 3.3.1-1
  Version table:
     3.3.2-2~bpo60+1 0
        100 http://backports.debian.org/debian-backports/ squeeze-backports/main i386 Packages
 *** 3.3.1-1 0
        500 http://ftp.de.debian.org/debian/ squeeze/main i386 Packages
        100 /var/lib/dpkg/status

> Does the same sa-update instance through a 404 error and then go onto the
> next mirror or are you saying you have to run it again?

That's a good question... I don't run it manually, I just see a cron error message almost every day (exactly the one that Patrice posted) and so I assumed that it failed.

> This sounds like a dupe of 6761 and in short, sa-update is designed to have
> some mirrors out of sync and it should try other servers.

I've looked into the sa-update code and it does seem to work exactly as you describe :)

I think it is very probable then that the update actually works fine every day, it just prints this confusing error message when it encounters the broken mirror. Maybe it should only print the error message for a failed mirror when all mirrors have failed?

Comment 6 Darxus 2012-09-25 14:30:54 UTC

(In reply to comment #5)
> broken mirror. Maybe it should only print the error message for a failed
> mirror when all mirrors have failed?

That's what I was thinking.  

Although it would be good to get some monitoring on those servers so we notice before the last one fails.

Comment 7 Kevin A. McGrail 2012-09-25 14:39:32 UTC

(In reply to comment #6)
> (In reply to comment #5)
> > broken mirror. Maybe it should only print the error message for a failed
> > mirror when all mirrors have failed?
> 
> That's what I was thinking.  
> 
> Although it would be good to get some monitoring on those servers so we
> notice before the last one fails.

PMC has monitoring reports on the servers so we know when they are down.  And to some extent, having failures like this is a good thing because it tests our resiliency.

Whether there is something to fix here is something I can debate but at a minimum, hiding the error as non-fatal for a single mirror is not bad but VERY low priority for me.

Comment 8 Martin von Wittich 2012-09-25 15:03:32 UTC

> hiding the error as non-fatal for a single mirror is not bad but
> VERY low priority for me.

That's understandable. Would it be possible though to remove the daryl.dostech.ca mirror from http://spamassassin.apache.org/updates/MIRRORED.BY until it is up-to-date again? That would spare us about 200 error mails per day :)

If not, is there another workaround? I could append "2>/dev/null" to the sa-update line in /etc/cron.daily/spamassassin, but that could mask other errors too. Modifying /etc/cron.daily/spamassassin to remove the daryl.dostech.ca mirror from /var/lib/spamassassin/3.003001/updates_spamassassin_org/MIRRORED.BY before it runs sa-update would work, wouldn't it?

Comment 9 Kevin A. McGrail 2012-09-25 15:24:56 UTC

(In reply to comment #8)
> > hiding the error as non-fatal for a single mirror is not bad but
> > VERY low priority for me.
> 
> That's understandable. Would it be possible though to remove the
> daryl.dostech.ca mirror from
> http://spamassassin.apache.org/updates/MIRRORED.BY until it is up-to-date
> again? That would spare us about 200 error mails per day :)
> 
> If not, is there another workaround? I could append "2>/dev/null" to the
> sa-update line in /etc/cron.daily/spamassassin, but that could mask other
> errors too. Modifying /etc/cron.daily/spamassassin to remove the
> daryl.dostech.ca mirror from
> /var/lib/spamassassin/3.003001/updates_spamassassin_org/MIRRORED.BY before
> it runs sa-update would work, wouldn't it?

Why are you checking more than 1x per day?

Comment 10 Darxus 2012-09-25 15:45:59 UTC

(In reply to comment #8)
> If not, is there another workaround? I could append "2>/dev/null" to the

Procmail filter it to /dev/null on that specific body.  Or to a folder so you can look at them if you want.

(In reply to comment #9)
> Why are you checking more than 1x per day?

"This problem affects all of our Debian servers"
Many servers.  Hopefully each once per day.

Comment 11 Darxus 2012-09-25 15:48:24 UTC

*** Bug 6761 has been marked as a duplicate of this bug. ***

Comment 12 Martin von Wittich 2012-09-25 16:18:14 UTC

(In reply to comment #10)
> (In reply to comment #9)
> > Why are you checking more than 1x per day?
> 
> "This problem affects all of our Debian servers"
> Many servers.  Hopefully each once per day.

Exactly, we are running about 600-700 Debian servers and all root mails are forwarded to a central support address. All of those servers run SpamAssassin :)

Comment 13 Kevin A. McGrail 2012-09-25 16:19:50 UTC

We'll likely add code to sa-update to see if we can hide the issue programatically.  But it's a low priority, sorry.

Comment 14 Darxus 2012-09-25 16:29:04 UTC

(In reply to comment #12)
> Exactly, we are running about 600-700 Debian servers and all root mails are

Fix it and send in a patch?  :)

Might be worth tracking down the code emitting the error and just commenting it out.

Comment 15 Kevin A. McGrail 2012-09-26 14:45:21 UTC

Based on the number of complaints, I've removed Daryl's mirror temporarily from the MIRRORED.BY file by updating the file on minotaur and then doing svn update on zones based on the rsyncd.conf file path for updates channel.

Comment 16 Martin von Wittich 2012-09-26 14:53:01 UTC

Thanks! If I find the time, I'll have a look into sa-update and see if I can come up with a patch.

Comment 17 Jarkko 2012-09-27 06:50:09 UTC

Created attachment 5092 [details]
Warn only with the verbose option

Comment 18 Kevin A. McGrail 2012-09-27 10:45:03 UTC

Comment on attachment 5092 [details]
Warn only with the verbose option

It's certainly the right line of code and helpful in that regard but the patch is too simplistic.  Those affected by the issue might wish to use it as a band-aid.

Here is my technical review:

First, the comment two lines above is now out of sync.

Second, is verbose turned on if debug is enabled?  If not, that would be another concern.

Third, there needs to be logic to display this information IF it is fatal and no update is able to be downloaded.

regards,
KAM

Comment 19 Darxus 2012-09-28 19:22:51 UTC

I've been wondering why I haven't seen this problem.  Is it because my server has IPv6 set up, and those seeing the error don't?  

(One person I talked to elsewhere who had this problem didn't have IPv6.)

Comment 20 Kevin A. McGrail 2012-09-28 19:25:43 UTC

(In reply to comment #19)
> I've been wondering why I haven't seen this problem.  Is it because my
> server has IPv6 set up, and those seeing the error don't?  
> 
> (One person I talked to elsewhere who had this problem didn't have IPv6.)

I believe I'm the only IPv6 and knock on wood, my mirror has been very stable.  We have a proposed second IPv6 mirror coming.

Regards,
KAM

Comment 21 Kevin A. McGrail 2012-09-29 03:35:37 UTC

(In reply to comment #15)
> Based on the number of complaints, I've removed Daryl's mirror temporarily
> from the MIRRORED.BY file by updating the file on minotaur and then doing
> svn update on zones based on the rsyncd.conf file path for updates channel.

I believe I also needed to run an svn update on people/minotaur in /www/spamassassin.apache.org/updates

In a few more hours, this should ripple through to the live server is my theory.

Comment 22 Larsen 2012-10-02 07:38:26 UTC

(In reply to comment #5)
> Maybe it should only print the error message for a failed
> mirror when all mirrors have failed?

+1

Comment 23 René Schwarz 2013-02-10 10:09:42 UTC

The behavior described by the original bug reporter now occurs for the mirror <http://sa-update.secnap.net/>, too. I get the following message since a few days:

/etc/cron.daily/spamassassin:
http: GET http://sa-update.secnap.net/1444342.tar.gz request failed: 404 Not Found: <html> <head><title>404 Not Found</title></head> <body bgcolor="white"> <center><h1>404 Not Found</h1></center> <hr><center>nginx/1.2.1</center> </body> </html>

Comment 24 Kevin A. McGrail 2013-02-11 14:26:39 UTC

*** Bug 6906 has been marked as a duplicate of this bug. ***

Comment 25 Kevin A. McGrail 2013-02-11 14:47:50 UTC

Re: Secnap mirror

I've disabled it, hopefully temporarily
[root@devel updates]# svn commit -m 'removed secnap mirror due to being out of sync - bug 6838'
Sending        updates/MIRRORED.BY
Transmitting file data .
Committed revision 1444801.

I've also opened a ticket about the issue formally with SecNap and will update progress.

Comment 26 Kevin A. McGrail 2013-02-11 15:24:18 UTC

(In reply to comment #25)
> Re: Secnap mirror
> 
> I've disabled it, hopefully temporarily
> [root@devel updates]# svn commit -m 'removed secnap mirror due to being out
> of sync - bug 6838'
> Sending        updates/MIRRORED.BY
> Transmitting file data .
> Committed revision 1444801.
> 
> I've also opened a ticket about the issue formally with SecNap and will
> update progress.

SecNap is working on the issue so I did not issue the full commands to implement the MIRRORED.BY.

Comment 27 René Schwarz 2013-02-11 15:56:00 UTC

(In reply to comment #26)
> SecNap is working on the issue so I did not issue the full commands to
> implement the MIRRORED.BY.

The mirror now seems to work again. Thanks for your fast response.

BTW: How is it possible to provide a mirror for sa-update? Unfortunately, I found no answer on the web for this question. So, if you need another mirror, let me know.

Comment 28 John Hardin 2013-02-11 16:20:54 UTC

(In reply to comment #27)
> 
> BTW: How is it possible to provide a mirror for sa-update? Unfortunately, I
> found no answer on the web for this question. So, if you need another
> mirror, let me know.

See bug 6876 for a discussion of automating this rather than relying on individual manually-managed mirror site donations.

Comment 29 Kevin A. McGrail 2013-02-11 17:24:22 UTC

(In reply to comment #27)
> (In reply to comment #26)
> > SecNap is working on the issue so I did not issue the full commands to
> > implement the MIRRORED.BY.
> 
> The mirror now seems to work again. Thanks for your fast response.
> 
> BTW: How is it possible to provide a mirror for sa-update? Unfortunately, I
> found no answer on the web for this question. So, if you need another
> mirror, let me know.

I'm not sure we need more mirrors because while these errors are annoying, updates should still work.  

However, the process should be documented and you can see http://wiki.apache.org/spamassassin/SaUpdateMirrorSetup now for more information.

Regards,
KAM

Comment 30 René Schwarz 2013-02-13 07:51:33 UTC

(In reply to comment #29)
> I'm not sure we need more mirrors because while these errors are annoying,
> updates should still work.  
> 
> However, the process should be documented and you can see
> http://wiki.apache.org/spamassassin/SaUpdateMirrorSetup now for more
> information.

I have set up a mirror at <http://sa-update.space-pro.be>. Just decide inside within your PMC if you need it or not and let me know your decision.


Kind regards,
René

Comment 31 Kevin A. McGrail 2013-06-18 03:49:53 UTC

Sorry for the delay and thanks for the offer to mirror.  I've updated the MIRRORED.BY to include your server and you should start seeing traffic soon.

Comment 32 René Schwarz 2013-06-18 05:16:43 UTC

(In reply to Kevin A. McGrail from comment #31)
> Sorry for the delay and thanks for the offer to mirror.  I've updated the
> MIRRORED.BY to include your server and you should start seeing traffic soon.

Hi Kevin,

you are welcome. If you experience any problems with my mirror, do not hesitate to contact me directly via e-mail.


Kind regards,
René

Comment 33 Tom Hendrikx 2013-07-04 07:44:31 UTC

I'm still seeing the secnap mirror causing errors as documented in this bug at least once a week. If secnap is working on this since February, maybe it is time to drop them from the mirrors files until the issue is resolved?

Comment 34 Mark Martinec 2013-07-04 11:30:54 UTC

(In reply to René Schwarz from comment #30)
> I have set up a mirror at <http://sa-update.space-pro.be>. Just decide
> inside within your PMC if you need it or not and let me know your decision.

I think it should become a requirement for new mirrors to offer
access over both protocol families (IPv4 and IPv6).
There seems to be plenty of former and just two(?) of the later kind.

Comment 35 René Schwarz 2013-07-05 07:31:47 UTC

(In reply to Mark Martinec from comment #34)
> I think it should become a requirement for new mirrors to offer
> access over both protocol families (IPv4 and IPv6).
> There seems to be plenty of former and just two(?) of the later kind.

The only mirror without IPv6 capability is now:

    #CONTACT: Wazir Shpoon and John Meyer
    http://sa-update.secnap.net/ weight=5

I have just updated my DNS servers to have a AAAA resource record for my mirror, so that it is now fully IPv6 capable (I've simply forgotten this when I set up the mirror).

Nevertheless, I think this is a good point.

Comment 36 Mark Martinec 2013-07-05 10:41:33 UTC

> I have just updated my DNS servers to have a AAAA resource record for my
> mirror, so that it is now fully IPv6 capable (I've simply forgotten this
> when I set up the mirror).

Great, much appreciated!

Comment 37 Tom Hendrikx 2013-07-05 11:19:02 UTC

While ipv6 support is great (yay!), this has nothing to do with the actual issue that this bug is about.

The error reported by sa-update for the secnap mirror is:

http: GET http://sa-update.secnap.net/1499267.tar.gz.sha1 request failed: 500 Server closed connection without sending any data back: Server closed connection without sending any data back at /usr/lib64/perl5/vendor_perl/5.12.4/Net/HTTP/Methods.pm line 345. 
channel: could not find working mirror, channel failed

Obviously there is an established connection to the server (be it either ipv4 or ipv6), but the server replies with an error response. This indicates a problem on the mirror, and has nothing to do with the used ip stack.

Comment 38 Kevin A. McGrail 2013-07-05 13:24:14 UTC

(In reply to Tom Hendrikx from comment #37)

> The only mirror without IPv6 capability is now:
> 
>     #CONTACT: Wazir Shpoon and John Meyer
>     http://sa-update.secnap.net/ weight=5

I'll update the mirror wiki and contact them about adding IPv6 if they can.

Regards,
KAM

Comment 39 Kevin A. McGrail 2013-07-08 19:35:40 UTC

(In reply to Kevin A. McGrail from comment #38)

> I'll update the mirror wiki and contact them about adding IPv6 if they can.

Wiki updated and I emailed the mirror operators to see if they can add an ipv6 stack.