[NUTCH-979] Add support for deleting Solr documents with ProtocolStatusCodes.NOTFOUND

• Type: New Feature
• Status: Closed
• Priority: Minor
• Resolution: Auto Closed
• Affects Version/s: nutchgora
• Fix Version/s: 2.5
• Component/s: indexer
• Patch Info: Patch Available
• Assignee: Unassigned
• Reporter: Markus Jelsma
• Created: 08/Apr/11 11:39
• Updated: 13/Oct/19 22:35
• Resolved: 13/Oct/19 22:35

Description

When issuing recrawls it can happen that certain urls have expired (i.e. URLs that don't exist anymore and return 404).
This issue creates a new command in the indexer that scans for WebPages with ProtocolStatusCodes.NOTFOUND and issues delete commands to Solr.

Issue Links

• incorporates: NUTCH-987 Support HTTP auth for Solr communication
• incorporates: NUTCH-1036 Solr jobs should increment counters in Reporter
• is related to: NUTCH-963 Add support for deleting Solr documents with STATUS_DB_GONE in CrawlDB (404 urls)
• relates to: NUTCH-1000 Add option not to commit to Solr