NoRobotsClient does not follow the standard for robot.txt location

I see that the url for the robots was relative to the path and this not follow the robots standard.

...
public static URL findRobotsUrl(URL base, String prefix) throws MalformedURLException {
URL url = new URL(base, "robots.txt");
boolean exist = existUrl(url);
...

It should be "new URL(base, "/robots.txt");"

I found this on the web:

• http://www.robotstxt.org/norobots-rfc.txt (sec 3.1)
• http://en.wikipedia.org/wiki/Robots.txt
• http://www.w3.org/TR/html4/appendix/notes.html#h-B.4.1.1

Attach a patch to solve this behavior.

Salu10.