Uploaded image for project: 'Nutch'
  1. Nutch
  2. NUTCH-2409

Injector: complete command-line help and counters

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Trivial
    • Resolution: Fixed
    • Affects Version/s: 1.13
    • Fix Version/s: 1.14
    • Component/s: injector
    • Labels:
      None
    • Patch Info:
      Patch Available

      Description

      See discussion in NUTCH-2335:

      • add counters for removed items from CrawlDb:
        Injector: Total urls removed from CrawlDb by filters: 2
        Injector: Total urls with status gone removed from CrawlDb (db.update.purge.404): 0
        
      • add -Ddb.update.purge.404=true to command-line help:
        Usage: Injector [-D...] <crawldb> <url_dir> [-overwrite|-update] [-noFilter] [-noNormalize] [-filterNormalizeAll]
        ...
         -D...          set or overwrite configuration property (property=value)
         -Ddb.update.purge.404=true
                        remove URLs with status gone (404) from CrawlDb
        

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          sebastian-nagel opened a new pull request #215: NUTCH-2409 Injector: complete command-line help and counters
          URL: https://github.com/apache/nutch/pull/215

          • add counters for items removed from CrawlDb
          • add -Ddb.update.purge.404=true to command-line help

          ----------------------------------------------------------------
          This is an automated message from the Apache Git Service.
          To respond to the message, please log on GitHub and use the
          URL above to go to the specific comment.

          For queries about this service, please contact Infrastructure at:
          users@infra.apache.org

          Show
          githubbot ASF GitHub Bot added a comment - sebastian-nagel opened a new pull request #215: NUTCH-2409 Injector: complete command-line help and counters URL: https://github.com/apache/nutch/pull/215 add counters for items removed from CrawlDb add -Ddb.update.purge.404=true to command-line help ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: users@infra.apache.org
          Hide
          githubbot ASF GitHub Bot added a comment -

          sebastian-nagel closed pull request #215: NUTCH-2409 Injector: complete command-line help and counters
          URL: https://github.com/apache/nutch/pull/215

          ----------------------------------------------------------------
          This is an automated message from the Apache Git Service.
          To respond to the message, please log on GitHub and use the
          URL above to go to the specific comment.

          For queries about this service, please contact Infrastructure at:
          users@infra.apache.org

          Show
          githubbot ASF GitHub Bot added a comment - sebastian-nagel closed pull request #215: NUTCH-2409 Injector: complete command-line help and counters URL: https://github.com/apache/nutch/pull/215 ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: users@infra.apache.org
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Nutch-trunk #3453 (See https://builds.apache.org/job/Nutch-trunk/3453/)
          NUTCH-2409 Injector: complete command-line help and counters - add (snagel: https://github.com/apache/nutch/commit/9b4a9df26c5b92c82029d030a9bf72cda043209c)

          • (edit) src/java/org/apache/nutch/crawl/Injector.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Nutch-trunk #3453 (See https://builds.apache.org/job/Nutch-trunk/3453/ ) NUTCH-2409 Injector: complete command-line help and counters - add (snagel: https://github.com/apache/nutch/commit/9b4a9df26c5b92c82029d030a9bf72cda043209c ) (edit) src/java/org/apache/nutch/crawl/Injector.java

            People

            • Assignee:
              wastl-nagel Sebastian Nagel
              Reporter:
              wastl-nagel Sebastian Nagel
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development