[NUTCH-1052] Multiple deletes of the same URL using SolrClean

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 1.3, 1.4
• Fix Version/s: 1.4, nutchgora
• Component/s: indexer
• Patch Info: Patch Available
• Assignee: Julien Nioche
• Reporter: Tim Pease
• Created: 14/Jul/11 21:59
• Updated: 20/Dec/11 11:30
• Resolved: 30/Sep/11 14:10

Description

The SolrClean class does not keep track of purged URLs, it only checks the URL status for "db_gone". When run multiple times the same list of URLs will be deleted from Solr. For small, stable crawl databases this is not a problem. For larger crawls this could be an issue. SolrClean will become an expensive operation.

One solution is to add a "purged" flag in the CrawlDatum metadata. SolrClean would then check this flag in addition to the "db_gone" status before adding the URL to the delete list.

Another solution is to add a new state to the status field "db_gone_and_purged".

Either way, the crawl DB will need to be updated after the Solr delete has successfully occurred.

Issue Links

• is superceded by: NUTCH-1139 Indexer to delete documents