[NUTCH-2634] Some links marked as "nofollow" are followed anyway.

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.20
• Assignee: Unassigned
• Reporter: Gerard Bouchar
• Created: 14/Aug/18 07:47
• Updated: 13/Mar/24 14:51
• Resolved: 08/Jan/23 19:10

Description

In order to check if an outlink in an <a> tag can be followed, nutch checks whether the value of its rel attribute is the exact string string "nofollow".
However, the rel attribute can contain a list of link types, all of which should be respected.

So nutch rightfully doesn't follow a link like:

<a href='top-secret.html' rel="nofollow">DO NOT FOLLOW THIS LINK</a>

but wrongfully follows :

<a href='top-secret.html' rel="nofollow noreferrer">DO NOT FOLLOW THIS LINK</a>

Because of the code duplication in nutch's html parsers, this should be fixed in two places:

1. parse/html/DOMContentUtils.java
2. parse/tika/DOMContentUtils.java

Issue Links

• links to: GitHub Pull Request #751