[OOZIE-3578] MapReduce counters cannot be used over 120

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 5.1.0
• Fix Version/s: 5.3.0
• Component/s: core
• Assignee: Dénes Bodó
• Reporter: Dénes Bodó
• Created: 07/Jan/20 07:19
• Updated: 21/Jul/20 15:52
• Resolved: 08/Jan/20 12:37

Description

When we create a mapreduce action which then creates more than 120 counters then the following exception is thrown:

org.apache.hadoop.mapreduce.counters.Limits.checkCounters(Limits.java:101)
org.apache.hadoop.mapreduce.counters.Limits.incrCounters(Limits.java:108)
org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.addCounter(AbstractCounterGroup.java:78)
org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.addCounterImpl(AbstractCounterGroup.java:95)
org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.findCounterImpl(AbstractCounterGroup.java:123)
org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.findCounter(AbstractCounterGroup.java:113)
org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.findCounter(AbstractCounterGroup.java:130)
org.apache.hadoop.mapreduce.counters.AbstractCounters.findCounter(AbstractCounters.java:155)
org.apache.hadoop.mapreduce.TypeConverter.fromYarn(TypeConverter.java:264)
org.apache.hadoop.mapred.ClientServiceDelegate.getJobCounters(ClientServiceDelegate.java:383)
org.apache.hadoop.mapred.YARNRunner.getJobCounters(YARNRunner.java:859)
org.apache.hadoop.mapreduce.Job$8.run(Job.java:820)
org.apache.hadoop.mapreduce.Job$8.run(Job.java:817)
java.security.AccessController.doPrivileged(Native Method)
javax.security.auth.Subject.doAs(Subject.java:422)
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
org.apache.hadoop.mapreduce.Job.getCounters(Job.java:817)
org.apache.hadoop.mapred.JobClient$NetworkedJob.getCounters(JobClient.java:379)
org.apache.oozie.action.hadoop.MapReduceActionExecutor.end(MapReduceActionExecutor.java:252)
org.apache.oozie.command.wf.ActionEndXCommand.execute(ActionEndXCommand.java:183)
org.apache.oozie.command.wf.ActionEndXCommand.execute(ActionEndXCommand.java:62)
org.apache.oozie.command.XCommand.call(XCommand.java:291)
org.apache.oozie.command.wf.ActionCheckXCommand.execute(ActionCheckXCommand.java:244)
org.apache.oozie.command.wf.ActionCheckXCommand.execute(ActionCheckXCommand.java:56)
org.apache.oozie.command.XCommand.call(XCommand.java:291)
java.util.concurrent.FutureTask.run(FutureTask.java:266)
org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:210)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)

It turned out if we use Oozie with Hadoop 3 the MR class called Limits is not initialised properly but with default values: https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/Limits.java#L40

If we set the "mapreduce.job.counters.max" to 500 in mapred-site.xml or in core-site.xml has no positive effect.

Issue Links

• is duplicated by: OOZIE-3604 Oozie Luncher & Map-Reduce Action Complete Successfully However Oozie Still Fails the Action