PerHost Crawling Policy ( crawl.ignore.external.links )

• Key: NUTCH-173
• Type: New Feature
• Status: Closed
• Resolution: Fixed
• Priority: Minor
• Assignee: Unassigned
• Reporter: Philippe EUGENE
• Component/s: fetcher
• Affects Version/s: 0.7, 0.7.1, 0.8

Description

There is two major way of crawl in Nutch.

Intranet Crawl : forbidden all, allow somes few host

Whole-web crawl : allow all, forbidden few thinks

I propose a third type of crawl.

Directory Crawl : The purpose of this crawl is to manage few thousands of host wihtout managing rules pattern in UrlFilterRegexp.

I made two patch for : 0.7, 0.7.1 and 0.8-dev

I propose a new boolean property in nutch-site.xml : crawl.ignore.external.links, with false value at default.
By default this new feature don't modify the behavior of nutch crawler.

When you setup this property to true, the crawler don't fetch external links of the host.
So the crawl is limited to the host that you inject at the beginning at the crawl.

I know there is some proposal of new crawl policy using the CrawlDatum in 0.8-dev branch.
This feature colud be a easiest way to add quickly new crawl feature to nutch, waiting for a best way to improve crawl policy.

I post two patch.
Sorry for my very poor english
–
Philippe