[NUTCH-1403] Add default ScoringFilter for manipulating metadata

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Implemented
• Fix Version/s: 1.19
• Assignee: Unassigned
• Reporter: Julien Nioche
• Created: 19/Jun/12 12:56
• Updated: 13/Mar/24 14:50
• Resolved: 01/Feb/21 20:25

Description

This is currently done by the urlmeta plugin, which has too vague a name and a redundant indexing filter now that we have the index-metadata plugin. This scoring filter would help defining which metadata to pass from :

• the crawl metadata to the content metadata
• the content metadata to the parse metadata
• the parse metadata to the crawldatum for the outlinks
  I'd make this scoring filter available by default i.e. not in a separate plugin as its functionalities are commonly used.

Issue Links

• links to: GitHub Pull Request #458