[NUTCH-802] Problems managing outlinks with large url length

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Won't Fix
• Fix Version/s: 1.7
• Component/s: parser
• Labels: nutch outlink parse parseoutputformat
• Patch Info: Patch Available
• Assignee: Andrzej Bialecki
• Reporter: Pablo Aragón
• Created: 18/Mar/10 10:39
• Updated: 22/May/13 03:53
• Resolved: 30/Apr/13 09:27

Description

Nutch can get idle during the collection of outlinks if the URL address of the outlink is too large.

The maximum sizes of an URL for the main web servers are:

• Apache: 4,000 bytes
• Microsoft Internet Information Server (IIS): 16, 384 bytes
• Perl HTTP::Daemon: 8.000 bytes

URL adress sizes bigger than 4000 bytes are problematic, so the limit should be set in the nutch-default.xml configuration file.

I attached a patch