Uploaded image for project: 'Droids'
  1. Droids
  2. DROIDS-103

incorrect robots.txt path

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • None
    • core
    • None

    Description

      droids currently attempts to locate the robots.txt file relative to the current access url. According to http://www.robotstxt.org/robotstxt.html robots.txt should always be accessed the root path: "/robots.txt".

      Attachments

        1. robots.txt.patch
          0.9 kB
          Paul Rogalinski

        Issue Links

          blocks

          Improvement - An improvement or enhancement to an existing feature or task. DROIDS-105 missing caching for robots.txt

          • Major - Major loss of function.
          • Open
          is related to

          Bug - A problem which impairs or prevents the functions of the product. DROIDS-4 NoRobotsClient does not follow the standard for robot.txt location

          • Major - Major loss of function.
          • Closed

          Activity

            People

              Unassigned Unassigned
              pulsar256 Paul Rogalinski
              Votes:
              1 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: