[NUTCH-2012] Merge parsechecker and indexchecker

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.10
• Fix Version/s: 1.15
• Assignee: Sebastian Nagel
• Reporter: Sebastian Nagel
• Created: 15/May/15 13:50
• Updated: 01/Oct/19 14:29
• Resolved: 11/Apr/18 12:21

Description

ParserChecker and IndexingFiltersChecker have evolved from simple tools to check parsers and parsefilters resp. indexing filters to powerful tools which emulate the crawling of a single URL/document:

• check robots.txt (NUTCH-2002)
• follow redirects (NUTCH-2004)

Keeping both tools in sync takes extra work (cf. NUTCH-1757/NUTCH-2006, also NUTCH-2002, NUTCH-2004 are done only for parsechecker). It's time to merge them

• either into one general debugging tool, keeping parsechecker and indexchecker as aliases
• centralize common code in one utility class

Issue Links

• incorporates: NUTCH-2145 parse/index checker fail to fetch valid percent-encoded URLs
• incorporates: NUTCH-2554 parserchecker can't fetch some URLs
• links to: GitHub Pull Request #310
• links to: GitHub Pull Request #348