[IMPALA-10057] TransactionKeepalive NoClassDefFoundError floods logs during JDBC_TEST/FE_TEST - ASF JIRA

Attach files

Attach Screenshot

Voters

Watch issue

Watchers

Create sub-task

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: Impala 4.0.0
Fix Version/s: Impala 3.4.2, Impala 4.2.0, Impala 4.1.1
Component/s: Infrastructure
Labels:
- flaky

Epic Color:
ghx-label-3

Description

For the both the normal tests and the docker-based tests, the Impala logs generated during the FE_TEST/JDBC_TEST can be huge:

$ du -c -h fe_test/ee_tests
4.0K	fe_test/ee_tests/minidumps/statestored
4.0K	fe_test/ee_tests/minidumps/impalad
4.0K	fe_test/ee_tests/minidumps/catalogd
16K	fe_test/ee_tests/minidumps
352K	fe_test/ee_tests/profiles
81G	fe_test/ee_tests
81G	total

Creating a tarball of these logs takes 10 minutes. The Impalad/catalogd logs are filled with this error over and over:

E0903 02:25:39.453887 12060 TransactionKeepalive.java:137] Unexpected exception thrown
Java exception follows:
java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: org/apache/impala/common/TransactionKeepalive$HeartbeatContext
	at org.apache.impala.common.TransactionKeepalive$DaemonThread.run(TransactionKeepalive.java:114)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoClassDefFoundError: org/apache/impala/common/TransactionKeepalive$HeartbeatContext
	... 2 more
Caused by: java.lang.ClassNotFoundException: org.apache.impala.common.TransactionKeepalive$HeartbeatContext
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	... 2 more

Two interesting points:

The frontend/jdbc tests are passing, so all of these errors in the impalad logs are not impacting tests.
These errors aren't concurrently with any of the other tests (ee tests, custom cluster tests, etc).

This is happening on normal core runs (including the GVO job that does FE_TEST/JDBC_TEST) on both Ubuntu and Centos 7. It is also happening on docker-based tests. A theory is that FE_TEST/JDBC_TEST have an Impala cluster running and then invoke maven to run the tests. Maven could manipulate jars while Impala is running. Maybe there is a race-condition or conflict when manipulating those jars that could cause the NoClassDefFoundError. It makes no sense for Impala not to be able to find TransactionKeepalive$HeartbeatContext.

When it happens, it is in a tight loop, printing the message more than once per millisecond. It fills the ERROR, WARNING, and INFO logs with that message, sometimes for multiple Impalads and/or catalogd.