Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
-
None
Description
I see that the url for the robots was relative to the path and this not follow the robots standard.
...
public static URL findRobotsUrl(URL base, String prefix) throws MalformedURLException {
URL url = new URL(base, "robots.txt");
boolean exist = existUrl(url);
...
It should be "new URL(base, "/robots.txt");"
I found this on the web:
- http://www.robotstxt.org/norobots-rfc.txt (sec 3.1)
- http://en.wikipedia.org/wiki/Robots.txt
- http://www.w3.org/TR/html4/appendix/notes.html#h-B.4.1.1
Attach a patch to solve this behavior.
Salu10.
Attachments
Attachments
Issue Links
- relates to
-
DROIDS-103 incorrect robots.txt path
- Closed