[NUTCH-2652] Fetcher launches more fetch tasks than fetch lists

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.15
• Fix Version/s: 1.16
• Component/s: fetcher

Environment

Hadoop, distributed mode (cluster of 22 nodes), CDH 5.15.1, Nutch built on recent master.

Seen the first time right now, although running since two months with Nutch 1.15. But the constraints causing inputs to be split may change from run to run.

• Assignee: Sebastian Nagel
• Reporter: Sebastian Nagel
• Created: 15/Oct/18 10:27
• Updated: 28/Jan/21 13:55
• Resolved: 20/Oct/18 17:40

Description

Fetcher may launch more fetcher tasks than there are fetch lists:

18/10/15 07:27:26 INFO input.FileInputFormat: Total input paths to process : 128
18/10/15 07:27:26 INFO mapreduce.JobSubmitter: number of splits:187

That's one design principle of Nutch as a MapRecude-based crawler: to ensure politeness and a guaranteed delay between requests to the same host/domain/ip all items of one host/domain/ip are put by Generator into the same fetch list. A fetch list may not be split because that would violate the politeness constraints - multiple fetcher tasks processing the splits of one fetch list then may send requests to the same host/domain/ip in parallel. See ab's chapter about Nutch in Hadoop the definitive guide (3rd edition).

Issue Links

• is caused by: NUTCH-2375 Upgrade the code base from org.apache.hadoop.mapred to org.apache.hadoop.mapreduce
• links to: GitHub Pull Request #394