Details
Description
For debugging or archival purposes it may be useful to let Fetcher store the robots.txt response (content and HTTP status). Of course, this should be optional and not by default.
Attachments
Issue Links
- links to
For debugging or archival purposes it may be useful to let Fetcher store the robots.txt response (content and HTTP status). Of course, this should be optional and not by default.