[NUTCH-2125] Metrics tool for relevancy

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.10
• Component/s: tool
• Labels: memex
• Assignee: Unassigned
• Reporter: Kim Whitehall
• Created: 28/Sep/15 15:05
• Updated: 28/Sep/15 15:11

Description

Purpose: a metric for determining if the “relevancy” of a crawl after each round and the “relevancy” of a page. NB: this is not a scoring plugin. By default, the first 25 terms will be stored.

• Return the topN terms per a page

• Return the topN terms per a segment based on tf-idf

• Leverage Apache Lucene libs