[NUTCH-2003] topN is not work correctly

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Auto Closed
• Affects Version/s: 2.3
• Fix Version/s: 2.5
• Assignee: Unassigned
• Reporter: Talat Uyarer
• Created: 29/Apr/15 09:32
• Updated: 22/Nov/19 15:46
• Resolved: 22/Nov/19 15:45

Description

I want to crawl top 1000 urls which are ordered by scores from webpage table. It doesnt work correctly.

When I use topN parameter, it is divided by map task counts (topN/ maptaskcounts = maptasktopN) Every map tasks generate maptasktopN urls of map tasks. Assume as I have 25 map tasks and I set topN parameter as 1000 and maptasktopN is calculated as 40. As Result We dont have top 1000 highest scored urls, we have 1000 urls of generated 40 highest scored urls per 25 map tasks.