[COMDEV-222] projects.a.o does not handle SVN problems well

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Projects Tool
• Assignee: Unassigned
• Reporter: Sebb
• Created: 20/Jun/17 13:51
• Updated: 18/Aug/17 22:12

Description

Recently there was some kind of a problem accessing SVN. This caused lots of errors [1] like:

Error when processing doap file http://svn.apache.org/repos/asf/xmlgraphics/site/trunk/content/fop/doap.rdf:
------------------------------------------------------------
Traceback (most recent call last):
File "parseprojects.py", line 154, in <module>
rdf = urllib.request.urlopen(url, timeout=URL_TIMEOUT).read()
File "/usr/lib/python3.4/urllib/request.py", line 161, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib/python3.4/urllib/request.py", line 469, in open
response = meth(req, response)
File "/usr/lib/python3.4/urllib/request.py", line 579, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python3.4/urllib/request.py", line 507, in error
return self._call_chain(*args)
File "/usr/lib/python3.4/urllib/request.py", line 441, in _call_chain
result = func(*args)
File "/usr/lib/python3.4/urllib/request.py", line 587, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden
URL: 'http://svn.apache.org/repos/asf/xmlgraphics/site/trunk/content/fop/doap.rdf'

Each such failure causes a message to site-dev
The failed gets also cause the doaps to be dropped:

Deleting obsolete file projects/xmlgraphics-fop.json

The script needs to be able to distinguish different kinds of errors and handle them accordingly.

For example, missing DOAPs are quite frequent, and should be retried next run.
However SVN errors such as the above should probably cause the run to be abandoned.

[1] https://reporter.apache.org/logs/2017-06/parseprojects_2017-06.log