[NUTCH-1971] The crawldb.url.filters property is not present in any configuration file

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.9
• Component/s: crawldb
• Labels: configuration crawldb nutch-default.xml
• Assignee: Unassigned
• Reporter: Luis Lopez
• Created: 19/Mar/15 22:51
• Updated: 28/Jan/21 14:03
• Resolved: 15/May/20 17:30

Description

In CrawlDbFilter.java there is a line for getting a boolean that sets if the filters are going to be applied or not:

public static final String URL_FILTERING = "crawldb.url.filters";

However in nutch-default.xml that property is not present. Currently the only way to set this value is using the -filter parameter from the command line.

The same applies to:
public static final String URL_NORMALIZING = "crawldb.url.normalizers";
public static final String URL_NORMALIZING_SCOPE = "crawldb.url.normalizers.scope";