[NUTCH-2328] GeneratorJob does not generate anything on second run

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Auto Closed
• Affects Version/s: 2.2, 2.3, 2.2.1, 2.3.1
• Fix Version/s: 2.5
• Component/s: generator
• Labels: fails generator subsequent

Environment

Ubuntu 16.04 / Hadoop 2.7.1

• Patch Info: Patch Available
• Flags: Patch, Important
• Assignee: Unassigned
• Reporter: Arthur B
• Created: 18/Oct/16 13:16
• Updated: 13/Oct/19 22:35
• Resolved: 13/Oct/19 22:35

Description

Given a topN parameter (ie 10) the GeneratorJob will fail to generate anything new on the subsequent runs within the same process space.
To reproduce the issue submit the GeneratorJob twice one after another to the M/R framework. Second time will say it generated 0 URLs.
This issue is due to the usage of the static count field (org.apache.nutch.crawl.GeneratorReducer#count) to determine if the topN value has been reached.

Issue Links

• duplicates: NUTCH-2330 GeneratorJob does not generate anything on second run