[NUTCH-235] Duplicate Inlink values

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.8
• Fix Version/s: 0.8
• Assignee: Andrzej Bialecki
• Reporter: Andrzej Bialecki
• Created: 19/Mar/06 03:19
• Updated: 21/Mar/06 07:22
• Resolved: 21/Mar/06 07:22

Description

Reading the code for LinkDb.reduce(): if we have page duplicates in input segments, or if we have two copies of the same input segment, we will create the same Inlink values (satisfying Inlink.equals()) multiple times. Since Inlinks is a facade for List, and not a Set, we will get duplicate Inlink-s in Inlinks (if you know what I mean .

The problem is easy to test: create a new linkdb based on 2 identical segments. This problem also makes it more difficult to properly implement LinkDB updating mechanism (i.e. incremental invertlinks).

I propose to change Inlinks to use a Set semantics, either explicitly by using a HashSet or implicitly by checking if a value to be added already exists. If there are no objections I'll commit this change shortly.