Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
Impala 4.0.0
-
ghx-label-3
Description
For the both the normal tests and the docker-based tests, the Impala logs generated during the FE_TEST/JDBC_TEST can be huge:
$ du -c -h fe_test/ee_tests 4.0K fe_test/ee_tests/minidumps/statestored 4.0K fe_test/ee_tests/minidumps/impalad 4.0K fe_test/ee_tests/minidumps/catalogd 16K fe_test/ee_tests/minidumps 352K fe_test/ee_tests/profiles 81G fe_test/ee_tests 81G total
Creating a tarball of these logs takes 10 minutes. The Impalad/catalogd logs are filled with this error over and over:
E0903 02:25:39.453887 12060 TransactionKeepalive.java:137] Unexpected exception thrown Java exception follows: java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: org/apache/impala/common/TransactionKeepalive$HeartbeatContext at org.apache.impala.common.TransactionKeepalive$DaemonThread.run(TransactionKeepalive.java:114) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.NoClassDefFoundError: org/apache/impala/common/TransactionKeepalive$HeartbeatContext ... 2 more Caused by: java.lang.ClassNotFoundException: org.apache.impala.common.TransactionKeepalive$HeartbeatContext at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ... 2 more
Two interesting points:
- The frontend/jdbc tests are passing, so all of these errors in the impalad logs are not impacting tests.
- These errors aren't concurrently with any of the other tests (ee tests, custom cluster tests, etc).
This is happening on normal core runs (including the GVO job that does FE_TEST/JDBC_TEST) on both Ubuntu and Centos 7. It is also happening on docker-based tests. A theory is that FE_TEST/JDBC_TEST have an Impala cluster running and then invoke maven to run the tests. Maven could manipulate jars while Impala is running. Maybe there is a race-condition or conflict when manipulating those jars that could cause the NoClassDefFoundError. It makes no sense for Impala not to be able to find TransactionKeepalive$HeartbeatContext.
When it happens, it is in a tight loop, printing the message more than once per millisecond. It fills the ERROR, WARNING, and INFO logs with that message, sometimes for multiple Impalads and/or catalogd.
Attachments
Issue Links
- is duplicated by
-
IMPALA-9900 High rate log spam during FE tests
- Resolved
-
IMPALA-10707 TestIceberg::test_iceberg_query failed with INNER EXCEPTION
- Resolved