[NUTCH-1270] some of Deflate encoded pages not fetched

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Cannot Reproduce
• Affects Version/s: 1.4
• Component/s: protocol
• Labels: fetch processDeflateEncoded

Environment

software

• Patch Info: Patch Available
• Assignee: Unassigned
• Reporter: behnam nikbakht
• Created: 08/Feb/12 10:37
• Updated: 28/Jan/21 14:04
• Resolved: 11/Jun/14 15:43

Description

it is a problem with some of web pages that fetched but their content can not retrived
after this change, this error fixed
we change lib-http/src/java/org/apache/nutch/protocol/http/api/HttpBase.java
public byte[] processDeflateEncoded(byte[] compressed, URL url) throws IOException {

if (LOGGER.isTraceEnabled())

{ LOGGER.trace("inflating...."); }

byte[] content = DeflateUtils.inflateBestEffort(compressed, getMaxContent());
+ if(content==null)
+ content = DeflateUtils.inflateBestEffort(compressed, 200000);

if (content == null)
throw new IOException("inflateBestEffort returned null");

if (LOGGER.isTraceEnabled())

{ LOGGER.trace("fetched " + compressed.length + " bytes of compressed content (expanded to " + content.length + " bytes) from " + url); }

return content;
}

Issue Links

• is related to: NUTCH-1736 Can't fetch page if http response header contains Transfer-Encoding：chunked