[NUTCH-2737] Generator: count and log reason of rejections during selection

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Implemented
• Affects Version/s: 1.15
• Fix Version/s: 1.16
• Component/s: generator
• Patch Info: Patch Available
• Assignee: Sebastian Nagel
• Reporter: Sebastian Nagel
• Created: 30/Sep/19 08:01
• Updated: 21/Jun/22 10:53
• Resolved: 01/Oct/19 14:19

Description

During the map phase of the selection step, the generator rejects many (usually most of) items for various reasons:

• not yet time for a refetch (returned by the fetch scheduler)
• generator score too low
• status does not match restrict status
• Jexl expression not matched

and some more. It would be useful if the reasons are counted and logged, esp. when the CrawlDb gets bigger and multiple options to restrict the selection are used.

Issue Links

• causes: NUTCH-2951 Crawl datum with metadata WRITABLE_GENERATE_TIME_KEY awaits fetching forever
• links to: GitHub Pull Request #477