[NUTCH-1044] Redirected URLs and possibly all of their outlinked URLs have invalid scores.

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.3
• Fix Version/s: 1.4
• Component/s: fetcher, parser
• Assignee: Julien Nioche
• Reporter: Nutch User - 1
• Created: 13/Jul/11 08:59
• Updated: 20/Dec/11 11:30
• Resolved: 10/Aug/11 19:55

Description

1.: http://lucene.472066.n3.nabble.com/URL-redirection-and-zero-scores-td3085311.html
2.: http://lucene.472066.n3.nabble.com/A-possible-solution-to-my-URL-redirection-and-zero-scores-problem-td3162164.html

Please note that also URLs redirected by meta refresh redirection do have invalid scores. For such URLs a CrawlDatum is created on the lines 157-177 of ParseOutputFormat.java (http://svn.apache.org/viewvc/nutch/branches/branch-1.3/src/java/org/apache/nutch/parse/ParseOutputFormat.java?view=markup). The new CrawlDatum's score isn't set anywhere after the creation so it's 1.0f as can be seen on the line 122 of CrawlDatum.java (http://svn.apache.org/viewvc/nutch/branches/branch-1.3/src/java/org/apache/nutch/crawl/CrawlDatum.java?view=markup).

It's another question whether the redirected URL's score should be just passed to the new URL or should the redirection be considered as a link in which case the new URL's score would be 'originalScore' / ('numberOfOutlinks' + 1).