[IMPALA-10057] TransactionKeepalive NoClassDefFoundError floods logs during JDBC_TEST/FE_TEST

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 4.0.0
• Fix Version/s: Impala 3.4.2, Impala 4.2.0, Impala 4.1.1
• Component/s: Infrastructure
• Labels: flaky
• Epic Color: ghx-label-3
• Assignee: Joe McDonnell
• Reporter: Joe McDonnell
• Created: 06/Aug/20 18:29
• Updated: 07/Apr/24 10:32
• Resolved: 10/May/22 03:08

Description

For the both the normal tests and the docker-based tests, the Impala logs generated during the FE_TEST/JDBC_TEST can be huge:

 

$ du -c -h fe_test/ee_tests
4.0K	fe_test/ee_tests/minidumps/statestored
4.0K	fe_test/ee_tests/minidumps/impalad
4.0K	fe_test/ee_tests/minidumps/catalogd
16K	fe_test/ee_tests/minidumps
352K	fe_test/ee_tests/profiles
81G	fe_test/ee_tests
81G	total

Creating a tarball of these logs takes 10 minutes. The Impalad/catalogd logs are filled with this error over and over:

E0903 02:25:39.453887 12060 TransactionKeepalive.java:137] Unexpected exception thrown
Java exception follows:
java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: org/apache/impala/common/TransactionKeepalive$HeartbeatContext
	at org.apache.impala.common.TransactionKeepalive$DaemonThread.run(TransactionKeepalive.java:114)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoClassDefFoundError: org/apache/impala/common/TransactionKeepalive$HeartbeatContext
	... 2 more
Caused by: java.lang.ClassNotFoundException: org.apache.impala.common.TransactionKeepalive$HeartbeatContext
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	... 2 more

Two interesting points:

1. The frontend/jdbc tests are passing, so all of these errors in the impalad logs are not impacting tests.
2. These errors aren't concurrently with any of the other tests (ee tests, custom cluster tests, etc).

This is happening on normal core runs (including the GVO job that does FE_TEST/JDBC_TEST) on both Ubuntu and Centos 7. It is also happening on docker-based tests. A theory is that FE_TEST/JDBC_TEST have an Impala cluster running and then invoke maven to run the tests. Maven could manipulate jars while Impala is running. Maybe there is a race-condition or conflict when manipulating those jars that could cause the NoClassDefFoundError. It makes no sense for Impala not to be able to find TransactionKeepalive$HeartbeatContext.

When it happens, it is in a tight loop, printing the message more than once per millisecond. It fills the ERROR, WARNING, and INFO logs with that message, sometimes for multiple Impalads and/or catalogd.

Issue Links

• is duplicated by: IMPALA-9900 High rate log spam during FE tests
• is duplicated by: IMPALA-10707 TestIceberg::test_iceberg_query failed with INNER EXCEPTION