[NUTCH-2381] In some situations the class TextProfileSignature gives different signatures for the same text "profile" page.

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.13
• Fix Version/s: 1.16
• Component/s: crawldb
• Labels: patch-available signature
• Patch Info: Patch Available
• Flags: Patch

Docs Text

My suggestion is change the methods compare of TokenComparator to:

  private static class TokenComparator implements Comparator<Token> {
    public int compare(Token t1, Token t2) {
           
      if (t2.cnt != t1.cnt) return t2.cnt - t1.cnt;
      return t2.val.compareTo(t1.val);

    }
  }

This way allows order by token frequency/name.

• Assignee: Sebastian Nagel
• Reporter: Rodrigo Joni Sestari
• Created: 02/May/17 11:53
• Updated: 28/Jan/21 13:56
• Resolved: 30/Sep/19 11:32

Description

In some situations the class TextProfileSignature gives different signatures for the same text "profile" page.

The method TextProfileSignature.calculate uses a HashMap to salve the tokens, after some process, the tokens come sorted by decreasing frequency.

For some pages like "http://curia.europa.eu/jcms/" the text "profile" is the same but the signature come different for each fetch.

Its happens because the tokens are sorted only by decreasing frequency. Tokens with the same frequency maybe not have the same order in different fetchs.

The HashMap no guarantees as to the order of the map and not guarantee that the order will remain constant over time.

My suggestion is change the methods TokenComparator.compare in order to sort by frequency and Name.

Rodrigo

Issue Links

• links to: GitHub Pull Request #473