Uploaded image for project: 'Nutch'
  1. Nutch
  2. NUTCH-1434

Indexer to delete robots noIndex

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 1.5.1
    • 1.6
    • indexer
    • None

    Description

      Nutch does not treat pages with meta robots="noindex" properly. All it does is remove the title and content fields from the parsed data. It does not stop those pages from being indexed, nor can it delete existing pages from the index if they change.

      Attachments

        1. NUTCH-1434-1.6-3.patch
          4 kB
          Markus Jelsma
        2. NUTCH-1434-1.6-2.patch
          6 kB
          Markus Jelsma
        3. NUTCH-1434-1.6-1.patch
          6 kB
          Markus Jelsma

        Issue Links

          is broken by

          Bug - A problem which impairs or prevents the functions of the product. NUTCH-1553 Property 'indexer.delete.robots.noindex' not working when using parser-html.

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Activity

            People

              markus17 Markus Jelsma
              markus17 Markus Jelsma
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: