[TEZ-4223] Adding new jars or resources after the first DAG runs does not work.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.10.0, 0.9.3
• Assignee: Harish JP
• Reporter: Harish JP
• Created: 10/Aug/20 06:20
• Updated: 25/Aug/20 15:00
• Resolved: 13/Aug/20 13:43

Description

If we executed DAG which needs additional jars after the first DAG is run, we get ClassNotFoundException.

 

 

2020-08-03 13:57:14,776 [INFO] [Dispatcher thread {Central}] |impl.DAGImpl|: Added additional resources : [[file:/dataroot/ycloud/yarn/nm/usercache/hive/appcache/application_1596442677646_0012/container_1596442677646_0012_01_000001/commons-pool-1.5.4.jar, file:/dataroot/ycloud/yarn/nm/usercache/hive/appcache/application_1596442677646_0012/container_1596442677646_0012_01_000001/postgresql-42.2.8.jar, file:/dataroot/ycloud/yarn/nm/usercache/hive/appcache/application_1596442677646_0012/container_1596442677646_0012_01_000001/hive-jdbc-handler-3.1.3000.7.2.2.0-73.jar, file:/dataroot/ycloud/yarn/nm/usercache/hive/appcache/application_1596442677646_0012/container_1596442677646_0012_01_000001/mssql-jdbc-6.2.1.jre7.jar, file:/dataroot/ycloud/yarn/nm/usercache/hive/appcache/application_1596442677646_0012/container_1596442677646_0012_01_000001/commons-dbcp-1.4.jar]] to classpath

org.apache.hive.com.esotericsoftware.kryo.KryoException: Unable to find class: org.apache.hive.storage.jdbc.JdbcInputFormat
Serialization trace:
inputFileFormatClass (org.apache.hadoop.hive.ql.plan.PartitionDesc)
aliasToPartnInfo (org.apache.hadoop.hive.ql.plan.MapWork)
        at org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:156)
        at org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:133)
...
...
Caused by: java.lang.ClassNotFoundException: org.apache.hive.storage.jdbc.JdbcInputFormat
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:583)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
        at java.base/java.lang.Class.forName0(Native Method)
        at java.base/java.lang.Class.forName(Class.java:398)
        at org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:154)
        ... 46 more