Uploaded image for project: 'IMPALA'
  2. IMPALA-10057

TransactionKeepalive NoClassDefFoundError floods logs during JDBC_TEST/FE_TEST



    • ghx-label-3


      For the both the normal tests and the docker-based tests, the Impala logs generated during the FE_TEST/JDBC_TEST can be huge:


      $ du -c -h fe_test/ee_tests
      4.0K	fe_test/ee_tests/minidumps/statestored
      4.0K	fe_test/ee_tests/minidumps/impalad
      4.0K	fe_test/ee_tests/minidumps/catalogd
      16K	fe_test/ee_tests/minidumps
      352K	fe_test/ee_tests/profiles
      81G	fe_test/ee_tests
      81G	total

      Creating a tarball of these logs takes 10 minutes. The Impalad/catalogd logs are filled with this error over and over:

      E0903 02:25:39.453887 12060 TransactionKeepalive.java:137] Unexpected exception thrown
      Java exception follows:
      java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: org/apache/impala/common/TransactionKeepalive$HeartbeatContext
      	at org.apache.impala.common.TransactionKeepalive$DaemonThread.run(TransactionKeepalive.java:114)
      	at java.lang.Thread.run(Thread.java:748)
      Caused by: java.lang.NoClassDefFoundError: org/apache/impala/common/TransactionKeepalive$HeartbeatContext
      	... 2 more
      Caused by: java.lang.ClassNotFoundException: org.apache.impala.common.TransactionKeepalive$HeartbeatContext
      	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
      	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
      	... 2 more

      Two interesting points:

      1. The frontend/jdbc tests are passing, so all of these errors in the impalad logs are not impacting tests.
      2. These errors aren't concurrently with any of the other tests (ee tests, custom cluster tests, etc).

      This is happening on normal core runs (including the GVO job that does FE_TEST/JDBC_TEST) on both Ubuntu and Centos 7. It is also happening on docker-based tests. A theory is that FE_TEST/JDBC_TEST have an Impala cluster running and then invoke maven to run the tests. Maven could manipulate jars while Impala is running. Maybe there is a race-condition or conflict when manipulating those jars that could cause the NoClassDefFoundError. It makes no sense for Impala not to be able to find TransactionKeepalive$HeartbeatContext.

      When it happens, it is in a tight loop, printing the message more than once per millisecond. It fills the ERROR, WARNING, and INFO logs with that message, sometimes for multiple Impalads and/or catalogd.


        Issue Links



              joemcdonnell Joe McDonnell
              joemcdonnell Joe McDonnell
              0 Vote for this issue
              8 Start watching this issue