Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Duplicate
-
3.3.0
-
None
-
None
Description
Loading an auxiliary jar from a Hdfs location on a node manager fails with ClassNotFound Exception
2019-11-08 03:59:49,256 INFO org.apache.hadoop.util.ApplicationClassLoader: classpath: [] 2019-11-08 03:59:49,256 INFO org.apache.hadoop.util.ApplicationClassLoader: system classes: [java., javax.accessibility., javax.activation., javax.activity., javax.annotation., javax.annotation.processing., javax.crypto., javax.imageio., javax.jws., javax.lang.model., -javax.management.j2ee., javax.management., javax.naming., javax.net., javax.print., javax.rmi., javax.script., -javax.security.auth.message., javax.security.auth., javax.security.cert., javax.security.sasl., javax.sound., javax.sql., javax.swing., javax.tools., javax.transaction., -javax.xml.registry., -javax.xml.rpc., javax.xml., org.w3c.dom., org.xml.sax., org.apache.commons.logging., org.apache.log4j., -org.apache.hadoop.hbase., org.apache.hadoop., core-default.xml, hdfs-default.xml, mapred-default.xml, yarn-default.xml] 2019-11-08 03:59:49,257 INFO org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices failed in state INITED java.lang.ClassNotFoundException: org.apache.auxtest.AuxServiceFromHDFS at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:189) at org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:157) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxiliaryServiceWithCustomClassLoader.getInstance(AuxiliaryServiceWithCustomClassLoader.java:169) at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:270) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108) at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:321) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:478) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:936) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:1016)
Repro:
1. Prepare a custom auxiliary service jar and place it on hdfs [hdfs@yarndocker-1 yarn]$ cat TestShuffleHandler2.java package org; import org.apache.hadoop.yarn.server.api.AuxiliaryService; import org.apache.hadoop.yarn.server.api.ApplicationInitializationContext; import org.apache.hadoop.yarn.server.api.ApplicationTerminationContext; import java.nio.ByteBuffer; public class TestShuffleHandler2 extends AuxiliaryService { public static final String MAPREDUCE_TEST_SHUFFLE_SERVICEID = "test_shuffle2"; public TestShuffleHandler2() { super("testshuffle2"); } @Override public void initializeApplication(ApplicationInitializationContext context) { } @Override public void stopApplication(ApplicationTerminationContext context) { } @Override public synchronized ByteBuffer getMetaData() { return ByteBuffer.allocate(0); } } [hdfs@yarndocker-1 yarn]$ javac -d . -cp `hadoop classpath` TestShuffleHandler2.java [hdfs@yarndocker-1 yarn]$ jar cvf auxhdfs.jar org/ [hdfs@yarndocker-1 mapreduce]$ hadoop fs -mkdir /AUX [hdfs@yarndocker-1 mapreduce]$ hadoop fs -put /tmp/auxhdfs.jar /AUX [hdfs@yarndocker-1 mapreduce]$ hadoop fs -chmod 777 /AUX [hdfs@yarndocker-1 mapreduce]$ hadoop fs -chmod 600 /AUX/auxhdfs.jar [hdfs@yarndocker-1 mapreduce]$ hadoop fs -chown -R yarn:hadoop /AUX 2. Configure YARN NodeManager (yarn-site.xml) to pick from Hdfs <property> <name>yarn.nodemanager.aux-services</name> <value>auxhdfs</value> </property> <property> <name>yarn.nodemanager.aux-services.auxhdfs.class&lt;/name> <value>org.TestShuffleHandler2</value> </property> <property> <name>yarn.nodemanager.aux-services.auxhdfs.remote-classpath</name> <value>/AUX/auxhdfs.jar</value> </property>
YARN-9965 issue is applicable for branch-3.2 and trunk (with yarn.nodemanager.aux-services.auxhdfs.system-classes configured) where NM runs fine on first time and not on subsequent restart.
This issue is applicable for trunk where NM fails to start any attempt without yarn.nodemanager.aux-services.auxhdfs.system-classes. This was an optional config in branch-3.2. YARN-9075 has set the configured custom class as a system class by default which caused the ClassNotFoundException as this class won't be in system class loader.
AuxServices#getSystemClasses has to return Null if not configured
private static String[] getSystemClasses(AuxServiceRecord service, String className) { AuxServiceConfiguration serviceConf = service.getConfiguration(); if (serviceConf != null && serviceConf.getProperty(SYSTEM_CLASSES) != null) { return StringUtils.split(serviceConf.getProperty(SYSTEM_CLASSES)); } return null; }
Attachments
Issue Links
- is caused by
-
YARN-9075 Dynamically add or remove auxiliary services
- Resolved
- is duplicated by
-
YARN-11053 AuxService should not use class name as default system classes
- Resolved
-
YARN-11030 ClassNotFoundException when aux service class is loaded from customized classpath
- Resolved
- relates to
-
YARN-9965 Fix NodeManager failing to start on subsequent times when Hdfs Auxillary Jar is set
- Resolved