Uploaded image for project: 'Nutch'
  1. Nutch
  2. NUTCH-2550

Fetcher fails to follow redirects

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 1.15
    • Fix Version/s: 1.15
    • Component/s: fetcher
    • Labels:
      None

      Description

      As I detailed in this github comment, it appears that PR #221 broke redirects. The fetcher will repeatedly fetch the original url rather than the one it's supposed to be redirecting to until http.redirect.max is exceeded, and then end with STATUS_FETCH_GONE.

      I noticed this issue when I was trying to crawl a site with a 301 MOVED PERMANENTLY status code.

      Should be pretty easy to fix though: I was able to get redirects working again simply by inserting the code

      url = fit.url

      here and here.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                hansbrende Hans Brende
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: