[NUTCH-1752] cache robots.txt rules per protocol:host:port

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.8, 2.2.1
• Fix Version/s: 2.3, 1.9
• Component/s: protocol
• Patch Info: Patch Available
• Assignee: Sebastian Nagel
• Reporter: Sebastian Nagel
• Created: 09/Apr/14 09:00
• Updated: 13/Mar/24 14:51
• Resolved: 12/May/14 19:40

Description

HttpRobotRulesParser caches rules from robots.txt per "protocol:host" (before NUTCH-1031 caching was per "host" only). The caching should be per "protocol:host:port". In doubt, a request to a different port may deliver a different robots.txt.
Applying robots.txt rules to a combination of host, protocol, and port is common practice:
Norobots RFC 1996 draft does not mention this explicitly (could be derived from examples) but others do:

• Wikipedia: "each protocol and port needs its own robots.txt file"
• Google webmasters: "The directives listed in the robots.txt file apply only to the host, protocol and port number where the file is hosted."