[NUTCH-100] New plugin urlfilter-db

• Type: Improvement
• Status: Closed
• Priority: Trivial
• Resolution: Won't Fix
• Affects Version/s: 0.8
• Component/s: fetcher

Environment

All Nutch versions

• Assignee: Unassigned
• Reporter: Gal Nitzan
• Created: 30/Sep/05 05:37
• Updated: 01/Apr/11 15:03
• Resolved: 01/Apr/11 15:03

Description

Hi,

I have written a new plugin, based on the URLFilter interface: urlfilter-db .

The purpose of this plugin is to filter domains, i.e. I would like to crawl the world but to fetch only certain domains.

The plugin uses a caching system (SwarmCache, easier to deploy than JCS) and on the back-end a database.

For each url
filter is called
end for

filter
get the domain name from url
call cache.get domain
if not in cache try the database
if in database cache it and return it
return null
end filter

The plugin reads the cache size, jdbc driver, connection string, table to use and domain field from nutch-site.xml