Fetcher improvements to parse and follow outlinks up to a specified depth. The number of outlinks to follow can be decreased by depth using a divisor. This patch introduces three new configuration directives:
Please, do not use this unless you know what you're doing. This feature does not consider the state of the CrawlDB nor does it consider generator settings such as limiting the number of pages per (domain|host|ip) queue. It is not polite to use this feature with high settings as it can fetch many pages from the same domain including duplicates.
Also, this feature will not work if fetcher.parse is disabled. With parsing enabled you might want to consider not to store downloaded content.