Uploaded image for project: 'Nutch'
  1. Nutch
  2. NUTCH-2409

Injector: complete command-line help and counters

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Trivial
    • Resolution: Fixed
    • Affects Version/s: 1.13
    • Fix Version/s: 1.14
    • Component/s: injector
    • Labels:
      None
    • Patch Info:
      Patch Available

      Description

      See discussion in NUTCH-2335:

      • add counters for removed items from CrawlDb:
        Injector: Total urls removed from CrawlDb by filters: 2
        Injector: Total urls with status gone removed from CrawlDb (db.update.purge.404): 0
        
      • add -Ddb.update.purge.404=true to command-line help:
        Usage: Injector [-D...] <crawldb> <url_dir> [-overwrite|-update] [-noFilter] [-noNormalize] [-filterNormalizeAll]
        ...
         -D...          set or overwrite configuration property (property=value)
         -Ddb.update.purge.404=true
                        remove URLs with status gone (404) from CrawlDb
        

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                wastl-nagel Sebastian Nagel
                Reporter:
                wastl-nagel Sebastian Nagel
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: