[NUTCH-289] CrawlDatum should store IP address

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 0.8
• Component/s: fetcher
• Assignee: Unassigned
• Reporter: Doug Cutting
• Created: 27/May/06 03:36
• Updated: 01/Apr/11 15:03
• Resolved: 01/Apr/11 15:03

Description

If the CrawlDatum stored the IP address of the host of it's URL, then one could:

• partition fetch lists on the basis of IP address, for better politeness;
• truncate pages to fetch per IP address, rather than just hostname. This would be a good way to limit the impact of domain spammers.

The IP addresses could be resolved when a CrawlDatum is first created for a new outlink, or perhaps during CrawlDB update.