[MAPREDUCE-7186] JobSubmitter#checkSpecs doesn not handle the situation when reduceNums is set less than zero.

• Type: Bug
• Status: Patch Available
• Priority: Minor
• Resolution: Unresolved
• Component/s: client
• Assignee: Hongyuan Li
• Reporter: Hongyuan Li
• Created: 21/Feb/19 09:11
• Updated: 02/Aug/19 05:54

Description

JobSubmitter#checkSpecs doesn not judge whether reduceNums is less than zero.

private void checkSpecs(Job job) throws ClassNotFoundException,
	InterruptedException, IOException {
	JobConf jConf = (JobConf)job.getConfiguration();
	// Check the output specification
	if (jConf.getNumReduceTasks() == 0 ?
	jConf.getUseNewMapper() : jConf.getUseNewReducer()) {
	org.apache.hadoop.mapreduce.OutputFormat<?, ?> output =
	ReflectionUtils.newInstance(job.getOutputFormatClass(),
	job.getConfiguration());
	output.checkOutputSpecs(job);
	} else {
	jConf.getOutputFormat().checkOutputSpecs(jtFs, jConf);
	}
	}
	

The code does handle the situation when reduceTaskNums is set to -1 or any other value less than zero.

When use hive, the "mapred.reduce.tasks" has a default value -1.

 

Issue Links

• duplicates: MAPREDUCE-7056 Ensure that mapreduce.job.reduces is not negative