Normalize Host during Generate

• Key: NUTCH-253
• Type: New Feature
• Status: Closed
• Resolution: Fixed
• Priority: Major
• Assignee: Unassigned
• Reporter: Rod Taylor
• Component/s: fetcher
• Affects Version/s: 0.8
• Fix Version/s: 0.9.0

Description

Extend URL Normalizer to allow for normalizion of the Hostname during Generate. By default no rules are applied.

In short, this allows foo.bar.com, bif.baz.bar.com and bar.com to be counted as being the same for generate.max.per.host if an appropriate regex is used.

Add "urlnormalizer-regex" to plugin.includes in nutch-site.xml in order to enable it.

Since several modules now extend the urlnormalizer base we use a "scope" parameter within plugin.xml to allow differentiation between the various urlnormalizer modules to select the right module for Generate.

Comments

[Andrzej Bialecki]

This issue is fixed as a part of NUTCH-365 in trunk. Changes were too intrusive to be ported to branch-0.8, although the patch in NUTCH-365 should apply more or less cleanly.