[NUTCH-1317] Max content length by MIME-type

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Auto Closed
• Affects Version/s: 1.4
• Fix Version/s: 2.5
• Component/s: parser
• Patch Info: Patch Available
• Assignee: Markus Jelsma
• Reporter: Markus Jelsma
• Created: 20/Mar/12 19:18
• Updated: 13/Oct/19 22:36
• Resolved: 13/Oct/19 22:36

Description

The good old http.content.length directive is not sufficient in large internet crawls. For example, a 5MB PDF file may be parsed without issues but a 5MB HTML file may time out.