Uploaded image for project: 'Nutch'
  1. Nutch
  2. NUTCH-2216

db.ignore.*.links to optionally follow internal redirects

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • 1.11
    • 1.14
    • fetcher
    • None

    Description

      db.ignore.internal.links doesn't follow any internal hyperlinks or redirects. Together with db.ignore.external.links it helps to restrict the crawl to a predefined set of URL's, for example provided by a customer.

      In many cases, a few of those URL's are redirects, which are not followed. This issue adds an option to optionally allow internal redirects despite db.ignore.internal.links being enabled.

      Attachments

        1. NUTCH-2216.patch
          4 kB
          Markus Jelsma
        2. NUTCH-2216.patch
          7 kB
          Markus Jelsma

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. NUTCH-2365 HTTP Redirects to SubDomains don't get crawled if db.ignore.external.links.mode == byDomain

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. NUTCH-2221 Introduce db.ignore.internal.links to FetcherThread

          • Major - Major loss of function.
          • Closed

          Task - A task that needs to be done. NUTCH-2220 Rename db.* options used only by the linkdb to linkdb.*

          • Major - Major loss of function.
          • Closed

          Activity

            People

              markus17 Markus Jelsma
              markus17 Markus Jelsma
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: