[NUTCH-49] Flag for generate to fetch only new pages to complement the -refetchonly flag

• Type: New Feature
• Status: Closed
• Priority: Minor
• Resolution: Won't Fix
• Component/s: fetcher
• Assignee: Unassigned
• Reporter: Luke Baker
• Created: 21/Apr/05 22:30
• Updated: 22/May/13 03:54
• Resolved: 12/Jan/13 20:27

Description

It would be useful, especially for research/testing purposes, to have a flag for the FetchListTool that make sure to only include URLs in the fetchlist that have not already been fetched (according to the information from the webdb that you're generating the fetchlist from).