[NUTCH-2524] bin/crawl: fix check for HostDb in distributed mode

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.15
• Fix Version/s: 1.15
• Component/s: bin
• Assignee: Unassigned
• Reporter: Semyon Semyonov
• Created: 06/Mar/18 10:09
• Updated: 01/Oct/19 14:28
• Resolved: 12/Mar/18 11:58

Description

In crawl script you can find something like 
if [[ -d "$CRAWL_PATH"/hostdb ]]; then
echo "Processing sitemaps based on hosts in HostDB"
__bin_nutch sitemap "$CRAWL_PATH"/crawldb -hostdb "$CRAWL_PATH"/hostdb -threads $NUM_THREADS
fi

if [[ -d "$CRAWL_PATH"/hostdb ]]; doesnt work for HDFS only for local mode.

Issue Links

• blocks: NUTCH-2510 Crawl script modification. HostDb : generate, optional usage and description
• links to: GitHub Pull Request #291