[NUTCH-2992] Fetcher: always block fetch queues when exceptions threshold is reached

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.19
• Fix Version/s: 1.20
• Component/s: fetcher
• Patch Info: Patch Available
• Assignee: Sebastian Nagel
• Reporter: Sebastian Nagel
• Created: 16/May/23 15:29
• Updated: 13/Mar/24 14:50
• Resolved: 23/May/23 14:25

Description

When the exception threshold of a fetch queue is reached (defined by fetcher.max.exceptions.per.queue, NUTCH-769), the queue must be blocked, even if it is empty at this time. Otherwise, if the queue feeder is still alive, new items may be added to the queue later on. See FetchItemQueues.java, line 306 for the reason and NUTCH-2947 for a similar bug.

Issue Links

• links to: GitHub Pull Request #762