[NUTCH-2564] protocol-http throws an error when the content-length header is not a number

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Assignee: Unassigned
• Reporter: Gerard Bouchar
• Created: 10/Apr/18 08:44
• Updated: 28/Jan/21 14:03
• Resolved: 12/Jun/18 19:14

Description

When a server sends an invalid Content-Length header (one that is not a valid number) with a plain-text http body, browsers simply ignore it, but protocol-http has a strange approach: if the header is composed only of white spaces, it ignores it, but if it contains other characters, it throws an error, preventing us from doing anything with the page.

It should simply ignore invalid Content-Length headers.

 

Relevant code: https://github.com/apache/nutch/blob/master/src/plugin/protocol-http/src/java/org/apache/nutch/protocol/http/HttpResponse.java#L354-L359