[NUTCH-2409] Injector: complete command-line help and counters

• Type: Improvement
• Status: Closed
• Priority: Trivial
• Resolution: Fixed
• Affects Version/s: 1.13
• Fix Version/s: 1.14
• Component/s: injector
• Patch Info: Patch Available
• Assignee: Sebastian Nagel
• Reporter: Sebastian Nagel
• Created: 17/Aug/17 10:52
• Updated: 13/Mar/24 14:50
• Resolved: 11/Sep/17 10:55

Description

See discussion in NUTCH-2335:

• add counters for removed items from CrawlDb:

  Injector: Total urls removed from CrawlDb by filters: 2
  Injector: Total urls with status gone removed from CrawlDb (db.update.purge.404): 0
  

• add -Ddb.update.purge.404=true to command-line help:

  Usage: Injector [-D...] <crawldb> <url_dir> [-overwrite|-update] [-noFilter] [-noNormalize] [-filterNormalizeAll]
  ...
   -D...          set or overwrite configuration property (property=value)
   -Ddb.update.purge.404=true
                  remove URLs with status gone (404) from CrawlDb
  

Issue Links

• is related to: NUTCH-2335 Injector not to filter and normalize existing URLs in CrawlDb
• links to: GitHub Pull Request #215