[NUTCH-2365] HTTP Redirects to SubDomains don't get crawled if db.ignore.external.links.mode == byDomain

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.12
• Fix Version/s: 1.14
• Component/s: fetcher

Environment

Fedora 25

• Patch Info: Patch Available
• Assignee: Sebastian Nagel
• Reporter: Sriram Nookala
• Created: 09/Mar/17 15:01
• Updated: 13/Mar/24 14:51
• Resolved: 18/Dec/17 16:31

Description

Crawling a domain http://www.mercenarytrader.com which redirects to https://members.mercenarytrader.com which doesn't get followed by Nutch even though 'db.ignore.external.links' is set to 'true' and 'db.ignore.external.links.mode' is set to 'byDomain'.
The bug is in FetcherThread where the comparison is by host and not by domain

String origHost = new URL(urlString).getHost().toLowerCase();
> String newHost = new URL(newUrl).getHost().toLowerCase();
> if (ignoreExternalLinks) {
> if (!origHost.equals(newHost)) {
> if (LOG.isDebugEnabled())

{ > LOG.debug(" - ignoring redirect " + redirType + " from " > + urlString + " to " + newUrl > + " because external links are ignored"); > }

> return null;
> }
> }

Issue Links

• is related to: NUTCH-2069 Ignore external links based on domain
• is related to: NUTCH-2216 db.ignore.*.links to optionally follow internal redirects
• links to: GitHub Pull Request #264