PerHost Crawling Policy ( crawl.ignore.external.links )

• Key: NUTCH-173
• Type: New Feature
• Status: Closed
• Resolution: Fixed
• Priority: Minor
• Assignee: Unassigned
• Reporter: Philippe EUGENE
• Component/s: fetcher
• Affects Version/s: 0.7, 0.7.1, 0.8

Description

There is two major way of crawl in Nutch.

Intranet Crawl : forbidden all, allow somes few host

Whole-web crawl : allow all, forbidden few thinks

I propose a third type of crawl.

Directory Crawl : The purpose of this crawl is to manage few thousands of host wihtout managing rules pattern in UrlFilterRegexp.

I made two patch for : 0.7, 0.7.1 and 0.8-dev

I propose a new boolean property in nutch-site.xml : crawl.ignore.external.links, with false value at default.
By default this new feature don't modify the behavior of nutch crawler.

When you setup this property to true, the crawler don't fetch external links of the host.
So the crawl is limited to the host that you inject at the beginning at the crawl.

I know there is some proposal of new crawl policy using the CrawlDatum in 0.8-dev branch.
This feature colud be a easiest way to add quickly new crawl feature to nutch, waiting for a best way to improve crawl policy.

I post two patch.
Sorry for my very poor english
–
Philippe

Comments

[Philippe EUGENE]

Patch for 0.7 and 0.7.1 version

[Philippe EUGENE]

Patch for 0.8-dev version

[Doug Cutting]

Couldn't you instead use a prefix-urlfilter generated from your crawl seed?

[Philippe EUGENE]

I have more than 5.000 hosts in my directory. I'm not sure about crawl performance with more than 5.000 rules.
It's easier for me to just manage a boolean value in the nutch conf.
I know this is not the natural way of crawl with Nutch, but it could be interested for somes nutch's user.
The most important problem : scoring from external links is affected by this patch.

[Christophe Noel]

We are TENS of nutch users using this precious patch.

Most of nutch users are not making whole-web search engine (too much hardware needed) but are willing to develop dedicated search engines.

We crawl sometimes 1000, sometimes 25000 web servers and it really slow down the crawling with 25000 entries in prefix-urlfilter.

This patch is NEEDED !

Christophe Noël
CETIC
Belgium

[Doug Cutting]

+1, with a few modifications.

Can you please re-generate this against the current sources? This patch does not apply for me.

Also, the fromHost should only be computed if crawl.ignore.external.links is true.

Finally, please add an entry to conf/nutch-default.xml for the new parameter in your patch.

Thanks!

[Stefan Neufeind]

Applies fine and works for me on 0.7.2.

[Stefan Neufeind]

Here is the 08-patch, corrected to work against nightly from 2006-05-20.
Also fromHost is now only generated if really needed and nutch-default.xml is patched as well. By the way: Where should a property for "crawl" be located in the config-file? In the "fetcher"-section? In that case please somebody move it up/down or rename the property before including it in the dev-tree.

But could somebody please review it quickly? I'm not sure it's 100% correct. Still investigating on my side ...

[Andrzej Bialecki]

Patch applied to trunk/ . Thank you!