[NUTCH-1314] Impose a limit on the length of outlink target urls

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.4
• Patch Info: Patch Available
• Assignee: Lewis John McGibbney
• Reporter: Ferdy
• Created: 16/Mar/12 16:41
• Updated: 13/Mar/24 14:50
• Resolved: 17/Jul/18 11:36

Description

In the past we have encountered situations where crawling specific broken sites resulted in ridiciously long urls that caused the stalling of tasks. The regex plugins (normalizing/filtering) processed single urls for hours, if not indefinitely hanging.

My suggestion is to limit the outlink url target length as soon possible. It is a configurable limit, the default is 3000. This should be reasonably long enough for most uses. But sufficienly strict enough to make sure regex plugins do not choke on urls that are too long. Please see attached patch for the Nutchgora implementation.

I'd like to hear what you think about this.

Issue Links

• is related to: NUTCH-1106 Options to skip url's based on length
• is related to: NUTCH-1531 URL filtering takes long time for very long URLs