[NUTCH-2376] Improve configurability of HTTP Accept* header fields

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.3.1, 1.13
• Fix Version/s: 2.4, 1.14
• Component/s: protocol
• Patch Info: Patch Available
• Assignee: Sebastian Nagel
• Reporter: Sebastian Nagel
• Created: 20/Apr/17 12:27
• Updated: 13/Mar/24 14:51
• Resolved: 19/May/17 10:56

Description

There should be no differences between protocol-http and protocol-httpclient whether the HTTP header fields Accept, Accept-Language and Accept-Charset are configurable. The configured values should be used for both plugins. In addition,

• it should be possible to unset the default values (overwrite with empty value) so that no HTTP header field is sent
• default values should be contained in nutch-default.xml

Note: Accept-Encoding should not be configurable as the protocol plugins must support the accepted compression codecs which may not be the case e.g. for Brotli.