[NUTCH-387] host normalization in Generator$Selector

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.9.0
• Component/s: generator

Environment

nutch trunk since revision 449088

• Assignee: Andrzej Bialecki
• Reporter: Johannes Zillmann
• Created: 18/Oct/06 09:40
• Updated: 03/Nov/06 11:55
• Resolved: 03/Nov/06 11:55

Description

the host normalization in Generator$Selector#reduce at line 177 seems broken:
String host = new URL(url.toString()).getHost();
...
try

{ host = normalizers.normalize(host, URLNormalizers.SCOPE_GENERATE_HOST_COUNT); host = new URL(host).getHost().toLowerCase(); }

catch (Exception e)

{ LOG.warn("Malformed URL: '" + host + "', skipping"); }

With default configuration the basic nomalizer will be called, which is doing 'new URL(host)'.
Also in line below 'new URL(host)' will be called.
Since url.getHost() always return the host without protocol, there will be a MalformedUrlException be thrown, always.
The job will continue as usual though, cause the exception is catched.