Uploaded image for project: 'Nutch'
  1. Nutch
  2. NUTCH-1708

use same id when indexing and deleting redirects

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.7
    • Fix Version/s: 2.3, 1.9
    • Component/s: indexer
    • Labels:
      None
    • Patch Info:
      Patch Available

      Description

      Redirect targets are indexed using "representative URL"

      • in Fetcher repr URL is determined by URLUtil.chooseRepr() and stored in CrawlDatum (CrawlDb). Repr URL is either source or target URL of the redirect pair.
      • NutchField "url" is filled by basic indexing filter with repr URL
      • id field used as unique key is filled from url per solrindex-mapping.xml

      Deletion of redirects is done in IndexerMapReduce.reduce() by key which is the URL of the redirect source. If the source URL is chosen as repr URL a redirect target may get erroneously deleted.

      Test crawl with seed http://wiki.apache.org/nutch which redirects to http://wiki.apache.org/nutch/. DummyIndexWriter (NUTCH-1707) indicates that same URL is deleted and added:

      delete  http://wiki.apache.org/nutch
      add     http://wiki.apache.org/nutch
      

        Attachments

        1. NUTCH-1708-2x-v1.patch
          2 kB
          Sebastian Nagel
        2. NUTCH-1708-trunk-v1.patch
          4 kB
          Sebastian Nagel
        3. NUTCH-1708-trunk-v2.patch
          5 kB
          Sebastian Nagel

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                snagel Sebastian Nagel
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: