[NUTCH-3020] ParseSegment should check for protocol's flags for truncation

• Type: Task
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 1.20
• Assignee: Unassigned
• Reporter: Tim Allison
• Created: 01/Nov/23 16:22
• Updated: 13/Mar/24 14:50
• Resolved: 06/Nov/23 20:07

Description

As discussed on the user list, several protocols can identify when a fetch has been truncated. ParseSegment only checks for the number of bytes fetched vs the http length header (if it exists). We should modify ParseSegment to check for notification of truncation from the protocols.

I noticed this specifically with okhttp, but other protocols may flag truncation as well.

Issue Links

• links to: GitHub Pull Request #794