Uploaded image for project: 'Nutch'
  1. Nutch
  2. NUTCH-1434

Indexer to delete robots noIndex

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.5.1
    • Fix Version/s: 1.6
    • Component/s: indexer
    • Labels:
      None

      Description

      Nutch does not treat pages with meta robots="noindex" properly. All it does is remove the title and content fields from the parsed data. It does not stop those pages from being indexed, nor can it delete existing pages from the index if they change.

        Attachments

        1. NUTCH-1434-1.6-1.patch
          6 kB
          Markus Jelsma
        2. NUTCH-1434-1.6-2.patch
          6 kB
          Markus Jelsma
        3. NUTCH-1434-1.6-3.patch
          4 kB
          Markus Jelsma

          Issue Links

            Activity

              People

              • Assignee:
                markus17 Markus Jelsma
                Reporter:
                markus17 Markus Jelsma
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: