[NUTCH-2060] dedup is removing entries with status db_gone

• Type: Bug
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 1.9
• Component/s: crawldb
• Assignee: Unassigned
• Reporter: Steven Hayles
• Created: 07/Jul/15 07:55
• Updated: 02/Mar/16 22:36

Description

Using the standard bin/crawl script, Solr is never informed when a previously indexed document has been deleted.
"bin/nutch update" sets db_gone status in the crawl db for requests returning HTTP 404 status.
"bin/nutch dedup" remove entries with status db_gone from the crawl db .
As a result "bin/nutch clean" never sees the db_gone status, so does not inform Solr.