[TEZ-4228] TezClassLoader should be used in TezChild and for Configuration objects

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 0.10.0, 0.9.3
• Labels: 0.10_blocker
• Assignee: László Bodor
• Reporter: László Bodor
• Created: 01/Sep/20 10:21
• Updated: 03/Sep/20 21:27
• Resolved: 03/Sep/20 14:13

Description

While testing a 0.10.0 staging artifact, I found that there multiple issues of TEZ-3860/TEZ-4223, which should be fixed before 0.10.0 release. The issues came up while hive precommit tests:
http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-1311/11/tests

The failing hive test uses ADD JAR functionality:

ADD JAR ${system:maven.local.repository}/org/apache/hive/hive-it-test-serde/${system:hive.version}/hive-it-test-serde-${system:hive.version}.jar;

1. There are code paths, where Configuration object's classloader is used, so if we add resources to TezClassLoader, we should point Configuration objects to use them, otherwise, we'll face exception as this: syslog_bad.log .
The changes in TezUtils solved the problem for container mode as syslog_good.log

2. There are codepaths, which use thread context classloader, so calling TezClassLoader.setupTezClassLoader early in TezChild solves issues like this: syslog_bad_context_classloader.log

3. The Configuration issue is the same in Hive LLAP, which is solved in 1), example failure reproduced locally is: hive_llap.log.

4. The context classloader problem is also present in LLAP daemons, which is being solved by HIVE-24108

Issue Links

• is caused by: TEZ-3860 JDK9: ReflectionUtils may not use URLClassLoader
• relates to: HIVE-24108 AddToClassPathAction should use TezClassLoader