Uploaded image for project: 'Nutch'
  1. Nutch
  2. NUTCH-1932

Automatically remove orphaned pages

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • 1.13
    • 1.14
    • None
    • None
    • Patch Available

    Description

      Orphan scoring filter that determines whether a page has become orphaned, e.g. it has no more other pages linking to it. If a page hasn't been linked to after markGoneAfter seconds, the page is marked as gone and is then removed by an indexer. If a page hasn't been linked to after markOrphanAfter seconds, the page is removed from the CrawlDB.

      Note: if you have this plugin enabled you MUST make sure you visit 'almost' every URL at least once within the refetch interval. If you don't, non-orphans may be marked as orphan and get deleted! You can use NUTCH-2368 to make sure this is the case.

      Attachments

        1. NUTCH-1932.patch
          9 kB
          Markus Jelsma
        2. NUTCH-1932.patch
          13 kB
          Markus Jelsma
        3. NUTCH-1932.patch
          14 kB
          Markus Jelsma
        4. NUTCH-1932.patch
          3 kB
          Markus Jelsma
        5. NUTCH-1932.patch
          12 kB
          Markus Jelsma
        6. NUTCH-1932.patch
          18 kB
          Markus Jelsma
        7. NUTCH-1932.patch
          19 kB
          Markus Jelsma
        8. NUTCH-1932-add.patch
          2 kB
          Sebastian Nagel
        9. NUTCH-1932.patch
          23 kB
          Markus Jelsma
        10. NUTCH-1932.patch
          23 kB
          Markus Jelsma
        11. NUTCH-1932.patch
          23 kB
          Markus Jelsma
        12. NUTCH-1932.patch
          23 kB
          Markus Jelsma
        13. NUTCH-1932.patch
          23 kB
          Markus Jelsma
        14. NUTCH-1932.patch
          24 kB
          Markus Jelsma
        15. NUTCH-1932.patch
          24 kB
          Markus Jelsma
        16. NUTCH-1932.patch
          23 kB
          Markus Jelsma

        Issue Links

          requires

          Bug - A problem which impairs or prevents the functions of the product. NUTCH-1921 Optionally disable HTTP if-modified-since header

          • Major - Major loss of function.
          • Closed

          New Feature - A new feature of the product, which has yet to be developed. NUTCH-1913 LinkDB to implement db.ignore.external.links

          • Trivial - Cosmetic problem like misspelt words or misaligned text.
          • Closed
          links to

          Web Link GitHub Pull Request #211

          Activity

            People

              markus17 Markus Jelsma
              markus17 Markus Jelsma
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: