[IMPALA-11528] hive-exec.pom doesn't include UDAF class

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 4.1.0
• Fix Version/s: Impala 4.2.0, Impala 4.1.1
• Component/s: Catalog
• Epic Color: ghx-label-5
• Assignee: Steve Carlin
• Reporter: Gabor Kaszab
• Created: 24/Aug/22 09:30
• Updated: 12/Jan/23 15:27
• Resolved: 27/Sep/22 03:01

Description

For hive-exec we load "UDF" that excludes the UDAF class:
https://github.com/apache/impala/blob/master/java/shaded-deps/hive-exec/pom.xml#L102

As a result when a UDAF is being loaded to catalog we get a NoClassDefFoundException.

I0819 09:20:07.777845     1 HiveUdfLoader.java:63] Loading UDF 'eu.radoop.datahandler.hive.udf.GenericUDAFCorrelationMatrix' from file:/tmp/e5a348f5-753a-485a-b37d-2a1420b09df7.jar
I0819 09:20:07.780457     1 MetastoreEventsProcessor.java:700] Metastore event processing restarted. Last synced event id was updated from 902310 to 902310
I0819 09:20:07.780704     1 jni-util.cc:286] java.lang.NoClassDefFoundError: org/apache/hadoop/hive/ql/exec/UDAF
	at org.apache.hadoop.hive.ql.exec.FunctionUtils.getUDFClassType(FunctionUtils.java:157)
	at org.apache.impala.hive.executor.HiveUdfLoader.<init>(HiveUdfLoader.java:68)
	at org.apache.impala.hive.executor.HiveUdfLoader.createWithLocalPath(HiveUdfLoader.java:155)
	at org.apache.impala.hive.executor.HiveJavaFunctionFactoryImpl.create(HiveJavaFunctionFactoryImpl.java:47)
	at org.apache.impala.hive.executor.HiveJavaFunctionFactoryImpl.create(HiveJavaFunctionFactoryImpl.java:67)
	at org.apache.impala.catalog.CatalogServiceCatalog.loadJavaFunctions(CatalogServiceCatalog.java:1756)
	at org.apache.impala.catalog.CatalogServiceCatalog.invalidateDb(CatalogServiceCatalog.java:1862)
	at org.apache.impala.catalog.CatalogServiceCatalog.reset(CatalogServiceCatalog.java:1994)
	at org.apache.impala.service.JniCatalog.<init>(JniCatalog.java:166)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.ql.exec.UDAF
	at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	... 9 more
I0819 09:20:07.780738     1 status.cc:129] NoClassDefFoundError: org/apache/hadoop/hive/ql/exec/UDAF
CAUSED BY: ClassNotFoundException: org.apache.hadoop.hive.ql.exec.UDAF
    @           0xd99193
    @          0x162f6d9
    @           0xd5f447
    @           0xd2ba46
    @           0xc85b28
    @           0xbd6fd0
    @     0x7ff4f9bf7554
    @           0xc7ba86
E0819 09:20:07.781023     1 catalog.cc:87] NoClassDefFoundError: org/apache/hadoop/hive/ql/exec/UDAF
CAUSED BY: ClassNotFoundException: org.apache.hadoop.hive.ql.exec.UDAF
. Impalad exiting.

In Impla 3.4 we handled this exception gracefully but apaprently in 4.1 Catalog gets terminated by this.

Issue Links

• is related to: IMPALA-11841 Support Hive (generic) UDAFs