[NUTCH-247] robot parser to restrict.

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 0.8
• Fix Version/s: 1.0.0
• Component/s: fetcher
• Patch Info: Patch Available
• Assignee: Sami Siren
• Reporter: Stefan Groschupf
• Created: 12/Apr/06 09:46
• Updated: 10/Apr/09 12:29
• Resolved: 24/Feb/09 09:55

Description

If the agent name and the robots agents are not proper configure the Robot rule parser uses LOG.severe to log the problem but solve it also.
Later on the fetcher thread checks for severe errors and stop if there is one.

RobotRulesParser:

if (agents.size() == 0)

{ agents.add(agentName); LOG.severe("No agents listed in 'http.robots.agents' property!"); }

else if (!((String)agents.get(0)).equalsIgnoreCase(agentName))

{ agents.add(0, agentName); LOG.severe("Agent we advertise (" + agentName + ") not listed first in 'http.robots.agents' property!"); }

Fetcher.FetcherThread:
if (LogFormatter.hasLoggedSevere()) // something bad happened
break;

I suggest to use warn or something similar instead of severe to log this problem.

Issue Links

• is related to: NUTCH-258 Once Nutch logs a SEVERE log item, Nutch fails forevermore