[NUTCH-965] Skip parsing for truncated documents

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: nutchgora, 1.5
• Component/s: parser
• Patch Info: Patch Available
• Assignee: Lewis John McGibbney
• Reporter: Alexis
• Created: 08/Feb/11 17:37
• Updated: 25/Feb/12 04:29
• Resolved: 22/Feb/12 11:06

Description

The issue you're likely to run into when parsing truncated FLV files is described here:
http://www.mail-archive.com/[e-mail]/msg01880.html

The parser library gets stuck in infinite loop as it encounters corrupted data due to for example truncating big binary files at fetch time.