[NUTCH-2896] Protocol-okhttp: make connection pool configurable

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Implemented
• Affects Version/s: 1.18
• Fix Version/s: 1.19
• Component/s: plugin, protocol
• Patch Info: Patch Available
• Assignee: Sebastian Nagel
• Reporter: Sebastian Nagel
• Created: 21/Sep/21 13:27
• Updated: 13/Mar/24 14:51
• Resolved: 15/Aug/22 14:38

Description

OkHttp's ConnectionPool "holds up to 5 idle connections which will be evicted after 5 minutes of inactivity." A pool of this size is suitable for site crawls but not for larger crawls over many different sites / hosts.

Note: in the current version (4.9.1) the connection pool is implemented as a linked queue. In order to scale beyond pool sizes exceeding 1000 we need to use a set of clients each with its own connection pool.

Issue Links

• links to: GitHub Pull Request #697