[NUTCH-1614] Plugin to exclude URLs matching regex list from indexing - to enable crawl but do not index

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 2.2.1
• Component/s: indexer
• Labels: plugin
• Patch Info: Patch Available
• Assignee: Unassigned
• Reporter: Brian
• Created: 17/Jul/13 17:11
• Updated: 12/Jun/14 09:13

Description

Some pages we need to crawl (such as some main pages and different views of a main page) to get all the other pages, but we don't want to index those pages themselves. Therefore we cannot use the url filter approach.

This plugin uses a file containing regex strings (see included sample file). If one of the regex strings matches with an entire URL, that URL will be excluded form indexing.

The file to use is specified by the following property in nutch-site.xml:

<property>
<name>indexer.url.filter.exclude.regex.file</name>
<value>regex-indexer-exclude-urls.txt</value>
<description>
Holds the file name containing the regex strings. Any URL matching one of these strings will be excluded from indexing.
"#" indicates a comment line and will be ignored.
</description>
</property>

Issue Links

• duplicates: NUTCH-1300 Indexer to filter and normalize URL's