[TIKA-3543] Various improvements to http fetcher

• Type: Task
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: Tim Allison
• Created: 01/Sep/21 16:02
• Updated: 02/Sep/21 20:17

Description

Let users set maxRedirects.
Let users select which http headers they'd like to store in the metadata.
Record the "target" URL in the metadata – the last redirect for a given URL.
Let users select whether or not to spool to disk.
Allow users to set a max bytes for content to spool to disk.