[NUTCH-1502] Test for CrawlDatum state transitions

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.7, 2.2
• Fix Version/s: 1.9
• Component/s: crawldb
• Patch Info: Patch Available
• Assignee: Unassigned
• Reporter: Sebastian Nagel
• Created: 06/Dec/12 21:40
• Updated: 13/Mar/24 14:50
• Resolved: 15/Jul/14 09:18

Description

An exhaustive test to check the matrix of CrawlDatum state transitions (CrawlStatus in 2.x) would be useful to detect errors esp. for continuous crawls where the number of possible transitions is quite large. Additional factors with impact on state transitions (retry counters, static and dynamic intervals) are also tested.
The tests will help to address the NUTCH-578 and NUTCH-1245. See the latter for a first sketchy patch.

Issue Links

• relates to: NUTCH-1564 AdaptiveFetchSchedule: sync_delta forces immediate refetch for documents not modified
• relates to: NUTCH-1422 bypass signature comparison when a document is redirected