Uploaded image for project: 'Nutch'
  1. Nutch
  2. NUTCH-2300

Fetcher to optionally save robots.txt

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.13
    • Component/s: fetcher, protocol, segment
    • Labels:
      None
    • Patch Info:
      Patch Available
    • Flags:
      Patch

      Description

      For debugging or archival purposes it may be useful to let Fetcher store the robots.txt response (content and HTTP status). Of course, this should be optional and not by default.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                snagel Sebastian Nagel
                Reporter:
                snagel Sebastian Nagel
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: