Details
-
Sub-task
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
2.9.0
-
None
-
Reviewed
Description
Third-party plugins need to add their jars to ATS. Most of the times, isolation is not needed. However, there needs to be a way to specify the path. For now, the jars on that path can be added to default classloader.
Attachments
Attachments
- YARN-5233-trunk.001.patch
- 12 kB
- Li Lu
- YARN-5233-trunk.002.patch
- 12 kB
- Li Lu
- YARN-5233-trunk.003.patch
- 12 kB
- Li Lu
Issue Links
- is duplicated by
-
YARN-5232 Support for specifying a path for ATS plugin jars
- Resolved
Activity
Attach a patch to add ATS plugin specific classpath and load them at run time.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 26s | Docker mode activated. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
0 | mvndep | 0m 7s | Maven dependency ordering for branch |
+1 | mvninstall | 6m 38s | trunk passed |
+1 | compile | 2m 17s | trunk passed |
+1 | checkstyle | 0m 38s | trunk passed |
+1 | mvnsite | 1m 16s | trunk passed |
+1 | mvneclipse | 0m 38s | trunk passed |
+1 | findbugs | 2m 21s | trunk passed |
+1 | javadoc | 0m 55s | trunk passed |
0 | mvndep | 0m 8s | Maven dependency ordering for patch |
+1 | mvninstall | 1m 2s | the patch passed |
+1 | compile | 2m 12s | the patch passed |
+1 | javac | 2m 12s | the patch passed |
-1 | checkstyle | 0m 35s | hadoop-yarn-project/hadoop-yarn: The patch generated 1 new + 216 unchanged - 1 fixed = 217 total (was 217) |
+1 | mvnsite | 1m 8s | the patch passed |
+1 | mvneclipse | 0m 32s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | xml | 0m 2s | The patch has no ill-formed XML file. |
+1 | findbugs | 2m 43s | the patch passed |
+1 | javadoc | 0m 52s | the patch passed |
+1 | unit | 0m 23s | hadoop-yarn-api in the patch passed. |
+1 | unit | 2m 10s | hadoop-yarn-common in the patch passed. |
+1 | unit | 1m 20s | hadoop-yarn-server-timeline-pluginstorage in the patch passed. |
+1 | asflicense | 0m 16s | The patch does not generate ASF License warnings. |
29m 31s |
Subsystem | Report/Notes |
---|---|
Docker | Image:yetus/hadoop:e2f6409 |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12811229/YARN-5233-trunk.001.patch |
JIRA Issue | |
Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml |
uname | Linux 9326de8e8706 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
git revision | trunk / 127d2c7 |
Default Java | 1.8.0_91 |
findbugs | v3.0.0 |
checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/12049/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt |
Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/12049/testReport/ |
modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage U: hadoop-yarn-project/hadoop-yarn |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/12049/console |
Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
+1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 27s | Docker mode activated. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
0 | mvndep | 0m 12s | Maven dependency ordering for branch |
+1 | mvninstall | 10m 6s | trunk passed |
+1 | compile | 4m 7s | trunk passed |
+1 | checkstyle | 0m 50s | trunk passed |
+1 | mvnsite | 2m 2s | trunk passed |
+1 | mvneclipse | 0m 35s | trunk passed |
+1 | findbugs | 2m 29s | trunk passed |
+1 | javadoc | 1m 4s | trunk passed |
0 | mvndep | 0m 8s | Maven dependency ordering for patch |
+1 | mvninstall | 1m 6s | the patch passed |
+1 | compile | 2m 7s | the patch passed |
+1 | javac | 2m 7s | the patch passed |
+1 | checkstyle | 0m 38s | hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 216 unchanged - 1 fixed = 216 total (was 217) |
+1 | mvnsite | 1m 20s | the patch passed |
+1 | mvneclipse | 0m 31s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | xml | 0m 2s | The patch has no ill-formed XML file. |
+1 | findbugs | 2m 41s | the patch passed |
+1 | javadoc | 1m 0s | the patch passed |
+1 | unit | 0m 24s | hadoop-yarn-api in the patch passed. |
+1 | unit | 2m 9s | hadoop-yarn-common in the patch passed. |
+1 | unit | 1m 14s | hadoop-yarn-server-timeline-pluginstorage in the patch passed. |
+1 | asflicense | 0m 18s | The patch does not generate ASF License warnings. |
36m 26s |
Subsystem | Report/Notes |
---|---|
Docker | Image:yetus/hadoop:e2f6409 |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12811237/YARN-5233-trunk.002.patch |
JIRA Issue | |
Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml |
uname | Linux c52f55afdb57 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
git revision | trunk / b1674ca |
Default Java | 1.8.0_91 |
findbugs | v3.0.0 |
Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/12051/testReport/ |
modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage U: hadoop-yarn-project/hadoop-yarn |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/12051/console |
Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
minor comment:
this logic could be simplified to throw exception directly.
} catch (Exception e) { LOG.warn("Error loading plugin " + name, e); caught = e; } if (cacheIdPlugin == null) { throw new RuntimeException("No class defined for " + name, caught); }
One question is that the customClassLoader should be able to load regular plugin class too ?
Thanks jianhe for the review! I've fixed the exception handling logic. For the classloader logic, the customized classloader can still load classes in other classpaths, since Java will fall back to other parent classloaders when the customized classloader cannot find the class. As discussed in Java's documentation for ClassLoader:
The ClassLoader class uses a delegation model to search for classes and resources. Each instance of ClassLoader has an associated parent class loader. When requested to find a class or resource, a ClassLoader instance will delegate the search for the class or resource to its parent class loader before attempting to find the class or resource itself. The virtual machine's built-in class loader, called the "bootstrap class loader", does not itself have a parent but may serve as the parent of a ClassLoader instance.
Oh and, our ApplicationClassLoader will delegate the loadClass process to its parent after the load fails in its code:
if (c == null && !isSystemClass(name, systemClasses)) { // Try to load class from this classloader's URLs. Note that this is like // the servlet spec, not the usual Java 2 behaviour where we ask the // parent to attempt to load first. try { c = findClass(name); if (LOG.isDebugEnabled() && c != null) { LOG.debug("Loaded class: " + name + " "); } } catch (ClassNotFoundException e) { if (LOG.isDebugEnabled()) { LOG.debug(e); } ex = e; } } if (c == null) { // try parent c = parent.loadClass(name); if (LOG.isDebugEnabled() && c != null) { LOG.debug("Loaded class from parent: " + name + " "); } }
+1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 24s | Docker mode activated. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
0 | mvndep | 0m 9s | Maven dependency ordering for branch |
+1 | mvninstall | 7m 43s | trunk passed |
+1 | compile | 2m 35s | trunk passed |
+1 | checkstyle | 0m 42s | trunk passed |
+1 | mvnsite | 1m 24s | trunk passed |
+1 | mvneclipse | 0m 42s | trunk passed |
+1 | findbugs | 2m 30s | trunk passed |
+1 | javadoc | 1m 3s | trunk passed |
0 | mvndep | 0m 7s | Maven dependency ordering for patch |
+1 | mvninstall | 1m 15s | the patch passed |
+1 | compile | 2m 25s | the patch passed |
+1 | javac | 2m 25s | the patch passed |
+1 | checkstyle | 0m 38s | hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 216 unchanged - 1 fixed = 216 total (was 217) |
+1 | mvnsite | 1m 21s | the patch passed |
+1 | mvneclipse | 0m 35s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | xml | 0m 1s | The patch has no ill-formed XML file. |
+1 | findbugs | 3m 6s | the patch passed |
+1 | javadoc | 0m 59s | the patch passed |
+1 | unit | 0m 25s | hadoop-yarn-api in the patch passed. |
+1 | unit | 2m 15s | hadoop-yarn-common in the patch passed. |
+1 | unit | 1m 25s | hadoop-yarn-server-timeline-pluginstorage in the patch passed. |
+1 | asflicense | 0m 19s | The patch does not generate ASF License warnings. |
33m 2s |
Subsystem | Report/Notes |
---|---|
Docker | Image:yetus/hadoop:9560f25 |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12816463/YARN-5233-trunk.003.patch |
JIRA Issue | |
Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml |
uname | Linux cb1a5b00cb19 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
git revision | trunk / 04f6ebb |
Default Java | 1.8.0_91 |
findbugs | v3.0.0 |
Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/12208/testReport/ |
modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage U: hadoop-yarn-project/hadoop-yarn |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/12208/console |
Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
SUCCESS: Integrated in Hadoop-trunk-Commit #10057 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10057/)
YARN-5233. Support for specifying a path for ATS plugin jars. (jianhe: rev 8a9d293dd60f6d51e1574e412d40746ba8175fe1)
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/main/java/org/apache/hadoop/yarn/server/timeline/EntityGroupFSTimelineStore.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/test/java/org/apache/hadoop/yarn/server/timeline/TestEntityGroupFSTimelineStore.java
Looking through discussions in
YARN-4577, looks like we want something similar to it.