[HBASE-8417] Hadoop2 MR tests fail intermittently because of ArithmeticException in yarn nodemanager

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Cannot Reproduce
• Component/s: hadoop2, mapreduce, test
• Assignee: Unassigned
• Reporter: Jonathan Hsieh
• Created: 24/Apr/13 14:13
• Updated: 16/Jun/22 06:03
• Resolved: 02/Aug/13 17:31

Description

Two tests in a recent run failed due to this exception (in the same place in yarn).

org.apache.hadoop.hbase.mapreduce.TestTableMapReduce.testMultiRegionTable
org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat.testExcludeAllFromMinorCompaction

2013-04-24 06:05:15,745 WARN  [AsyncDispatcher event handler] resourcemanager.RMAuditLogger(255): USER=jenkins	OPERATION=Application Finished - Failed	TARGET=RMAppManager	RESULT=FAILURE	DESCRIPTION=App failed with state: FAILED	PERMISSIONS=Application application_1366808640917_0002 failed 1 times due to AM Container for appattempt_1366808640917_0002_000001 exited with  exitCode: -1000 due to: java.lang.ArithmeticException: / by zero
	at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:368)
	at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
	at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
	at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
	at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:279)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:851)

.Failing this attempt.. Failing the application.	APPID=application_1366808640917_0002

My guess is that this affects all the yarn/mr2/hadoop2 MR jobs.

Issue Links

• is broken by: MAPREDUCE-4940 division by zero in getLocalPathForWrite()
• is broken by: HADOOP-9079 LocalDirAllocator throws ArithmeticException