[YARN-11204] Various MapReduce tests fail with NPE in AggregatedLogDeletionService.stopRMClient

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: log-aggregation
• Labels: pull-request-available
• Target Version/s: 3.4.0
• Hadoop Flags: Reviewed
• Assignee: Szilard Nemeth
• Reporter: Szilard Nemeth
• Created: 29/Jun/22 10:00
• Updated: 28/Jan/24 05:15
• Resolved: 29/Jun/22 13:13

Description

During testing of HADOOP-15327, I noticed that lots of unit test are failing in the module called 'hadoop-mapreduce-client-jobclient'.
See this link for details: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3259/9/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client.txt

In case of the above Jenkins link expires later, I attached the same text file to this jira.

Let's see one example: 
org.apache.hadoop.mapred.TestMROpportunisticMaps#testAllOpportunisticMaps
Logs are also attached.

An example stacktrace, for reference: 

2022-06-29 11:24:13,510 INFO  [Listener at 0.0.0.0/8049] service.AbstractService (AbstractService.java:noteFailure(268)) - Service TestMROpportunisticMaps failed in state STOPPED
java.lang.NullPointerException
    at org.apache.hadoop.yarn.logaggregation.AggregatedLogDeletionService.stopRMClient(AggregatedLogDeletionService.java:322)
    at org.apache.hadoop.yarn.logaggregation.AggregatedLogDeletionService.serviceStop(AggregatedLogDeletionService.java:229)
    at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
    at org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:54)
    at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:102)
    at org.apache.hadoop.service.CompositeService.stop(CompositeService.java:160)
    at org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:134)
    at org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.serviceStop(JobHistoryServer.java:203)
    at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
    at org.apache.hadoop.mapreduce.v2.MiniMRYarnCluster$JobHistoryServerWrapper.serviceStop(MiniMRYarnCluster.java:293)
    at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
    at org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:54)
    at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:102)
    at org.apache.hadoop.service.CompositeService.stop(CompositeService.java:160)
    at org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:134)
    at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
    at org.apache.hadoop.mapred.MiniMRYarnClusterAdapter.stop(MiniMRYarnClusterAdapter.java:56)
    at org.apache.hadoop.mapred.TestMROpportunisticMaps.doTest(TestMROpportunisticMaps.java:108)
    at org.apache.hadoop.mapred.TestMROpportunisticMaps.doTest(TestMROpportunisticMaps.java:74)
    at org.apache.hadoop.mapred.TestMROpportunisticMaps.testAllOpportunisticMaps(TestMROpportunisticMaps.java:60)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
    at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
    at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
    at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
    at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
    at org.junit.runners.Suite.runChild(Suite.java:128)
    at org.junit.runners.Suite.runChild(Suite.java:27)
    at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
    at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
    at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
    at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
    at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
    at com.intellij.rt.junit.IdeaTestRunner$Repeater$1.execute(IdeaTestRunner.java:38)
    at com.intellij.rt.execution.junit.TestsRepeater.repeat(TestsRepeater.java:11)
    at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:35)
    at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:235)
    at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:54)

The cause of this NPE is YARN-11188 (commit link: https://github.com/apache/hadoop/commit/e6ecc4f3e4433ae23fd745f6e0c641a019664253).

The issue is that in org.apache.hadoop.yarn.logaggregation.AggregatedLogDeletionService#stopRMClient, there's an iteration over org.apache.hadoop.yarn.logaggregation.AggregatedLogDeletionService#tasks, that is only initialized in org.apache.hadoop.yarn.logaggregation.AggregatedLogDeletionService#scheduleLogDeletionTasks.

This method won't run if log aggregation is not enabled, but the AggregatedLogDeletionService is active.
An easy fix for this is to initialize org.apache.hadoop.yarn.logaggregation.AggregatedLogDeletionService#tasks to an empty list on the field level.

 

Issue Links

• links to: GitHub Pull Request #4513