[NUTCH-2510] Crawl script modification. HostDb : generate, optional usage and description

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Implemented
• Affects Version/s: 1.15
• Fix Version/s: 1.15
• Component/s: bin
• Assignee: Unassigned
• Reporter: Semyon Semyonov
• Created: 14/Feb/18 10:33
• Updated: 01/Oct/19 14:29
• Resolved: 02/Jul/18 14:28

Description

Script crawl now includes hostdb update as a part of crawling cycle, but :
1) There is no hostdb parameter for generate

2) Generation of hostdb is not optional, therefore hostdb is generated each step without asking of user. It should be an optional parameter.

3) Description of 1 and 2.

Issue Links

• is blocked by: NUTCH-2524 bin/crawl: fix check for HostDb in distributed mode
• links to: Pull request