diff --git hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeManagerAuxServiceClassPathIsolation.md hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeManagerAuxServiceClassPathIsolation.md new file mode 100644 index 0000000..a54fc0f --- /dev/null +++ hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeManagerAuxServiceClassPathIsolation.md @@ -0,0 +1,55 @@ + + +Using auxiliary service classpath isolation with YARN +======================= + + + +To launch auxiliary services on a NodeManager, users have to add their jar to NodeManager's classpath directly, thus put them on the system classloader. But if multiple versions of the plugin are present on the classpath, there is no control over which version actually gets loaded. Or if there are any conflicts between the dependencies introduced by the auxiliary services and the NodeManager itself, they can break the NodeManager, the auxiliary services, or both. To solve this issue, we can instantiate auxiliary services using a classloader that is different from the system classloader. + +Configuration +--------------------- + +This section describes the configuration variables for aux-service classpath isolation. + +The following settings need to be set in *yarn-site.xml*. + +|Configuration Name | Description | +|:---- |:---- | +| `yarn.nodemanager.aux-services.%s.classpath` | Provide local directory which includes the related jar file as well as all the dependencies’ jar file. We could specify the single jar file or use ${local_dir_to_jar}/* to load all jars under the dep directory. | +| `yarn.nodemanager.aux-services.%s.system-classes` | Normally, we do not need to set this configuration. The class would be loaded from customized classpath if it does not belongs to system-classes. For example, by default, the package org.apache.hadoop is in the system-classes, if your class CustomAuxService is in the package org.apache.hadoop, it would not be loaded from customized classpath. To solve this, either we could change the package for CustomAuxService, or configure our own system-classes which exclude org.apache.hadoop. | + +### Configuration Examples + + + yarn.nodemanager.aux-services + mapreduce_shuffle,CustomAuxService + + + + yarn.nodemanager.aux-services.CustomAuxService.classpath + ${local_dir_to_jar}/* + + + + yarn.nodemanager.aux-services.CustomAuxService.class + org.aux.CustomAuxService + + + + yarn.nodemanager.aux-services.mapreduce_shuffle.class + org.apache.hadoop.mapred.ShuffleHandler + +