[NUTCH-2929] Fetcher: start threads slowly to avoid that resources are temporarily exhausted

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Implemented
• Affects Version/s: 1.18
• Fix Version/s: 1.19
• Patch Info: Patch Available
• Assignee: Unassigned
• Reporter: Sebastian Nagel
• Created: 11/Jan/22 14:03
• Updated: 13/Mar/24 14:51
• Resolved: 14/Jan/22 09:42

Description

Fetcher spins all threads without any delay. This may cause that certain resources are temporarily exhausted if all threads start fetching the first pages simultaneously.

The issue has been observed by Tika warnings about overuse of the SAXParser pool which appeared only during the first 2-5 minutes of fetching a segment. See https://lists.apache.org/thread/lo6b9wdlxy2lz12wmosldgl9x9ov1cks - adding a short delay between thread launches makes the warnings disappear.

Issue Links

• links to: GitHub Pull Request #722