[TEZ-3860] JDK9: ReflectionUtils may not use URLClassLoader

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.10.0, 0.9.3
• Assignee: László Bodor
• Reporter: Zoltan Haindrich
• Created: 26/Oct/17 06:40
• Updated: 15/Dec/20 18:37
• Resolved: 18/Nov/19 19:28

Description

The following code
https://github.com/apache/tez/blob/master/tez-api/src/main/java/org/apache/tez/common/ReflectionUtils.java#L125

is not compatible with JDK9 since the classloader is an AppClassLoader

causes exceptions like this:

java.lang.ClassCastException: java.base/jdk.internal.loader.ClassLoaders$AppClassLoader cannot be cast to java.base/java.net.URLClassLoader
	at org.apache.tez.common.ReflectionUtils.addResourcesToSystemClassLoader(ReflectionUtils.java:125) ~[tez-api-0.9.0.jar:0.9.0]
	at org.apache.tez.dag.utils.RelocalizationUtils.addUrlsToClassPath(RelocalizationUtils.java:57) ~[tez-common-0.9.0.jar:0.9.0]
	at org.apache.tez.dag.app.dag.impl.DAGImpl$StartTransition.transition(DAGImpl.java:1793) ~[tez-dag-0.9.0.jar:0.9.0]
	at org.apache.tez.dag.app.dag.impl.DAGImpl$StartTransition.transition(DAGImpl.java:1776) ~[tez-dag-0.9.0.jar:0.9.0]
	at org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:362) ~[hadoop-yarn-common-2.8.1.jar:?]
	at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302) ~[hadoop-yarn-common-2.8.1.jar:?]
	at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46) ~[hadoop-yarn-common-2.8.1.jar:?]
	at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448) ~[hadoop-yarn-common-2.8.1.jar:?]
	at org.apache.tez.state.StateMachineTez.doTransition(StateMachineTez.java:59) ~[tez-dag-0.9.0.jar:0.9.0]
	at org.apache.tez.dag.app.dag.impl.DAGImpl.handle(DAGImpl.java:1156) [tez-dag-0.9.0.jar:0.9.0]
	at org.apache.tez.dag.app.dag.impl.DAGImpl.handle(DAGImpl.java:147) [tez-dag-0.9.0.jar:0.9.0]
	at org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher.handle(DAGAppMaster.java:2251) [tez-dag-0.9.0.jar:0.9.0]
	at org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher.handle(DAGAppMaster.java:2242) [tez-dag-0.9.0.jar:0.9.0]
	at org.apache.tez.common.AsyncDispatcher.dispatch(AsyncDispatcher.java:180) [tez-common-0.9.0.jar:0.9.0]
	at org.apache.tez.common.AsyncDispatcher$1.run(AsyncDispatcher.java:115) [tez-common-0.9.0.jar:0.9.0]
	at java.base/java.lang.Thread.run(Thread.java:844) [?:?]

Issue Links

• blocks: HIVE-17909 JDK9: Tez may not use URLClassloader
• causes: TEZ-4122 TestMRRJobsDAGApi should set TezClassLoader
• causes: TEZ-4228 TezClassLoader should be used in TezChild and for Configuration objects
• links to: GitHub Pull Request #51