Major For debugging or archival purposes it may be useful to let Fetcher store the robots.txt response (content and HTTP status). Of course, this should be optional and not by default.
- links to
For debugging or archival purposes it may be useful to let Fetcher store the robots.txt response (content and HTTP status). Of course, this should be optional and not by default.
