[DROIDS-4] NoRobotsClient does not follow the standard for robot.txt location

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Assignee: Unassigned
• Reporter: Javier Puerto
• Created: 12/Oct/08 12:26
• Updated: 24/Nov/10 20:37
• Resolved: 12/Oct/08 19:13

Description

I see that the url for the robots was relative to the path and this not follow the robots standard.

...
public static URL findRobotsUrl(URL base, String prefix) throws MalformedURLException {
URL url = new URL(base, "robots.txt");
boolean exist = existUrl(url);
...

It should be "new URL(base, "/robots.txt");"

I found this on the web:

• http://www.robotstxt.org/norobots-rfc.txt (sec 3.1)
• http://en.wikipedia.org/wiki/Robots.txt
• http://www.w3.org/TR/html4/appendix/notes.html#h-B.4.1.1

Attach a patch to solve this behavior.

Salu10.

Issue Links

• relates to: DROIDS-103 incorrect robots.txt path