Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-5080

test_java_udfs: OutOfMemoryError: PermGen space

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: Impala 2.9.0
    • Fix Version/s: Impala 2.9.0
    • Component/s: Frontend
    • Labels:

      Description

      There was a Java OOM related to PermGen space in a release-mode UDF test. Googling, it seems this is a memory area for Java, and that one gets around it by increasing limits or other tweaks.

      The UDF code hasn't changed in a while. Was there a recent change that used up too much memory, and as a side-effect this test can fail? Have we slowly been approaching this limit all along?

      end-to-end traceback:

       TestUdfExecution.test_java_udfs[exec_option: {'disable_codegen': True, 'exec_single_node_rows_threshold': 100, 'enable_expr_rewrites': True} | table_format: text/none]
      query_test/test_udfs.py:309: in test_java_udfs
          self.run_test_case('QueryTest/java-udf', vector, use_db=unique_database)
      common/impala_test_suite.py:359: in run_test_case
          result = self.__execute_query(target_impalad_client, query, user=user)
      common/impala_test_suite.py:567: in __execute_query
          return impalad_client.execute(query, user=user)
      common/impala_connection.py:160: in execute
          return self.__beeswax_client.execute(sql_stmt, user=user)
      beeswax/impala_beeswax.py:173: in execute
          handle = self.__execute_query(query_string.strip(), user=user)
      beeswax/impala_beeswax.py:339: in __execute_query
          self.wait_for_completion(handle)
      beeswax/impala_beeswax.py:359: in wait_for_completion
          raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
      E   ImpalaBeeswaxException: ImpalaBeeswaxException:
      E    Query aborted:
      E   OutOfMemoryError: PermGen space
      
      select * from
      (select max(int_col) from functional.alltypesagg
       where identity(bool_col) union all
      (select max(int_col) from functional.alltypesagg
       where identity(tinyint_col) > 1 union all
      (select max(int_col) from functional.alltypesagg
       where identity(smallint_col) > 1 union all
      (select max(int_col) from functional.alltypesagg
       where identity(int_col) > 1 union all
      (select max(int_col) from functional.alltypesagg
       where identity(bigint_col) > 1 union all
      (select max(int_col) from functional.alltypesagg
       where identity(float_col) > 1.0 union all
      (select max(int_col) from functional.alltypesagg
       where identity(double_col) > 1.0 union all
      (select max(int_col) from functional.alltypesagg
       where identity(string_col) > '1' union all
      (select max(int_col) from functional.alltypesagg
       where not identity(bool_col) union all
      (select max(int_col) from functional.alltypesagg
       where identity(tinyint_col) > 2 union all
      (select max(int_col) from functional.alltypesagg
       where identity(smallint_col) > 2 union all
      (select max(int_col) from functional.alltypesagg
       where identity(int_col) > 2 union all
      (select max(int_col) from functional.alltypesagg
       where identity(bigint_col) > 2 union all
      (select max(int_col) from functional.alltypesagg
       where identity(float_col) > 2.0 union all
      (select max(int_col) from functional.alltypesagg
       where identity(double_col) > 2.0 union all
      (select max(int_col) from functional.alltypesagg
       where identity(string_col) > '2'
      )))))))))))))))) v;
      

      from the impalad.log:

      I0316 10:57:33.113581  6342 jni-util.cc:176] java.lang.OutOfMemoryError: PermGen space
          at java.lang.ClassLoader.defineClass1(Native Method)
          at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
          at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
          at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
          at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
          at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
          at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
          at java.security.AccessController.doPrivileged(Native Method)
          at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
          at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
          at java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:789)
          at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
          at java.lang.Class.forName0(Native Method)
          at java.lang.Class.forName(Class.java:274)
          at org.apache.impala.hive.executor.UdfExecutor.init(UdfExecutor.java:584)
          at org.apache.impala.hive.executor.UdfExecutor.<init>(UdfExecutor.java:218)
      

        Attachments

          Activity

            People

            • Assignee:
              twmarshall Thomas Tauber-Marshall
              Reporter:
              mikesbrown Michael Brown
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: