[NUTCH-1419] parsechecker and indexchecker to report protocol status

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: nutchgora, 1.6
• Fix Version/s: 1.7, 2.2
• Component/s: indexer, parser
• Patch Info: Patch Available
• Assignee: Unassigned
• Reporter: Sebastian Nagel
• Created: 03/Jul/12 12:46
• Updated: 22/May/13 03:54
• Resolved: 26/Mar/13 19:43

Description

Parsechecker and indexchecker should report the protocol status when the fetch was not successful (status other than 200/ok).

In case of a redirect, the protocol status contains the URL a redirect points to. Usually, this URL should be checked instead of the original one which is not indexed. The content of a redirect response is less useful (and often empty):

% nutch indexchecker http://lucene.apache.org/nutch/
fetching: http://lucene.apache.org/nutch/
parsing: http://lucene.apache.org/nutch/
contentType: text/html
content :       301 Moved Permanently Moved Permanently The document has moved here . Apache/2.4.1 (Unix) OpenSSL/1.
title : 301 Moved Permanently
host :  lucene.apache.org
tstamp :        Tue Jul 03 13:27:32 CEST 2012
url :   http://lucene.apache.org/nutch/