[NUTCH-2637] Number of fetcher reducers is misconfigured when the arg not passed

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.3, 2.3.1
• Fix Version/s: 2.4
• Component/s: fetcher
• Assignee: Unassigned
• Reporter: Fumio Nakajima
• Created: 18/Aug/18 05:59
• Updated: 13/Mar/24 14:50
• Resolved: 12/Sep/18 09:06

Description

I'm kind a new to this, so sorry if i'm wrong.
The thing is the number of fetcher reducers are currently set to the value of "mapreduce.job.maps" when the arg not passed. It should be "mapreduce.job.reduces".

 

https://github.com/apache/nutch/blob/2.x/src/java/org/apache/nutch/fetcher/FetcherJob.java#L216

Line: 216, branch-2.X

if (numTasks == null || numTasks < 1) {
currentJob.setNumReduceTasks(currentJob.getConfiguration().getInt(
"mapreduce.job.maps", currentJob.getNumReduceTasks()));
} else {
currentJob.setNumReduceTasks(numTasks);
}

Issue Links

• links to: GitHub Pull Request #381