[NUTCH-2518] Must check return value of job.waitForCompletion()

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.15
• Fix Version/s: 1.15
• Component/s: crawldb, fetcher, generator, hostdb, linkdb
• Assignee: Unassigned
• Reporter: Sebastian Nagel
• Created: 05/Mar/18 11:16
• Updated: 01/Oct/19 14:29
• Resolved: 04/Apr/18 10:59

Description

The return value of job.waitForCompletion() of the new MapReduce API (NUTCH-2375) must always be checked. If it's not true, the job has been failed or killed. Accordingly, the program

• should not proceed with further jobs/steps
• must clean-up temporary data, unlock CrawlDB, etc.
• exit with non-zero exit value, so that scripts running the crawl workflow can handle the failure

Cf. NUTCH-2076, NUTCH-2442, NUTCH-2375 PR #221.

Issue Links

• is caused by: NUTCH-2375 Upgrade the code base from org.apache.hadoop.mapred to org.apache.hadoop.mapreduce
• supercedes: NUTCH-2161 Interrupted failed and/or killed tasks fail to clean up temp directories in HDFS
• supercedes: NUTCH-1783 Cleanup temp folders in case of failures
• links to: GitHub Pull Request #299
• links to: GitHub Pull Request #307