The return value of job.waitForCompletion() of the new MapReduce API (
NUTCH-2375) must always be checked. If it's not true, the job has been failed or killed. Accordingly, the program
- should not proceed with further jobs/steps
- must clean-up temporary data, unlock CrawlDB, etc.
- exit with non-zero exit value, so that scripts running the crawl workflow can handle the failure