Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-5080

test_java_udfs: OutOfMemoryError: PermGen space

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: Impala 2.9.0
    • Fix Version/s: Impala 2.9.0
    • Component/s: Frontend
    • Labels:

      Description

      There was a Java OOM related to PermGen space in a release-mode UDF test. Googling, it seems this is a memory area for Java, and that one gets around it by increasing limits or other tweaks.

      The UDF code hasn't changed in a while. Was there a recent change that used up too much memory, and as a side-effect this test can fail? Have we slowly been approaching this limit all along?

      end-to-end traceback:

       TestUdfExecution.test_java_udfs[exec_option: {'disable_codegen': True, 'exec_single_node_rows_threshold': 100, 'enable_expr_rewrites': True} | table_format: text/none]
      query_test/test_udfs.py:309: in test_java_udfs
          self.run_test_case('QueryTest/java-udf', vector, use_db=unique_database)
      common/impala_test_suite.py:359: in run_test_case
          result = self.__execute_query(target_impalad_client, query, user=user)
      common/impala_test_suite.py:567: in __execute_query
          return impalad_client.execute(query, user=user)
      common/impala_connection.py:160: in execute
          return self.__beeswax_client.execute(sql_stmt, user=user)
      beeswax/impala_beeswax.py:173: in execute
          handle = self.__execute_query(query_string.strip(), user=user)
      beeswax/impala_beeswax.py:339: in __execute_query
          self.wait_for_completion(handle)
      beeswax/impala_beeswax.py:359: in wait_for_completion
          raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
      E   ImpalaBeeswaxException: ImpalaBeeswaxException:
      E    Query aborted:
      E   OutOfMemoryError: PermGen space
      
      select * from
      (select max(int_col) from functional.alltypesagg
       where identity(bool_col) union all
      (select max(int_col) from functional.alltypesagg
       where identity(tinyint_col) > 1 union all
      (select max(int_col) from functional.alltypesagg
       where identity(smallint_col) > 1 union all
      (select max(int_col) from functional.alltypesagg
       where identity(int_col) > 1 union all
      (select max(int_col) from functional.alltypesagg
       where identity(bigint_col) > 1 union all
      (select max(int_col) from functional.alltypesagg
       where identity(float_col) > 1.0 union all
      (select max(int_col) from functional.alltypesagg
       where identity(double_col) > 1.0 union all
      (select max(int_col) from functional.alltypesagg
       where identity(string_col) > '1' union all
      (select max(int_col) from functional.alltypesagg
       where not identity(bool_col) union all
      (select max(int_col) from functional.alltypesagg
       where identity(tinyint_col) > 2 union all
      (select max(int_col) from functional.alltypesagg
       where identity(smallint_col) > 2 union all
      (select max(int_col) from functional.alltypesagg
       where identity(int_col) > 2 union all
      (select max(int_col) from functional.alltypesagg
       where identity(bigint_col) > 2 union all
      (select max(int_col) from functional.alltypesagg
       where identity(float_col) > 2.0 union all
      (select max(int_col) from functional.alltypesagg
       where identity(double_col) > 2.0 union all
      (select max(int_col) from functional.alltypesagg
       where identity(string_col) > '2'
      )))))))))))))))) v;
      

      from the impalad.log:

      I0316 10:57:33.113581  6342 jni-util.cc:176] java.lang.OutOfMemoryError: PermGen space
          at java.lang.ClassLoader.defineClass1(Native Method)
          at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
          at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
          at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
          at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
          at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
          at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
          at java.security.AccessController.doPrivileged(Native Method)
          at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
          at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
          at java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:789)
          at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
          at java.lang.Class.forName0(Native Method)
          at java.lang.Class.forName(Class.java:274)
          at org.apache.impala.hive.executor.UdfExecutor.init(UdfExecutor.java:584)
          at org.apache.impala.hive.executor.UdfExecutor.<init>(UdfExecutor.java:218)
      

        Activity

        Hide
        mikesbrown Michael Brown added a comment -

        Alexander Behm As FE/Java owner please triage/assign accordingly, and let me know if there could have been more in the initial bug description to help.

        Show
        mikesbrown Michael Brown added a comment - Alexander Behm As FE/Java owner please triage/assign accordingly, and let me know if there could have been more in the initial bug description to help.
        Hide
        mikesbrown Michael Brown added a comment -

        Hmm, see also IMPALA-5088.

        Show
        mikesbrown Michael Brown added a comment - Hmm, see also IMPALA-5088 .
        Hide
        srus@cloudera.com Silvius Rus added a comment -

        Alexander Behm, is this a duplicate of IMPALA-5088?

        Show
        srus@cloudera.com Silvius Rus added a comment - Alexander Behm , is this a duplicate of IMPALA-5088 ?
        Hide
        alex.behm Alexander Behm added a comment -

        Matthew Mulder, Matthew Jacobs, have you seen this pop up lately?

        Show
        alex.behm Alexander Behm added a comment - Matthew Mulder , Matthew Jacobs , have you seen this pop up lately?
        Hide
        mjacobs Matthew Jacobs added a comment -

        Yes, it happened last friday.

        Show
        mjacobs Matthew Jacobs added a comment - Yes, it happened last friday.
        Hide
        alex.behm Alexander Behm added a comment -

        Thanks. That confirms this not the same as IMPALA-5088.

        Show
        alex.behm Alexander Behm added a comment - Thanks. That confirms this not the same as IMPALA-5088 .
        Hide
        twmarshall Thomas Tauber-Marshall added a comment -

        commit 1fa331537acf4c2460f3787ba81a5a64cac89c99
        Author: Thomas Tauber-Marshall <tmarshall@cloudera.com>
        Date: Fri Apr 14 13:28:52 2017 -0700

        IMPALA-5080: OutOfMemory PermGen space

        The problem is that over time our usage of PermGen space has
        gradually gone up, recently hitting a point where we may run
        out during testing. The fix is to increase the allowed max
        perm gen space.

        The default is 64mb on 32bit systems and 82mb on 64bit systems.
        After bumping it up to 128mb, I haven't seen any failures on my
        machine.

        I also verified that this is not a leak by running the test,
        test_java_udfs, 10 times, checking the perm gen usage with jmap,
        running it a further 300 times, and checking that the perm gen
        usage remained the same.

        Change-Id: Iccfb69cdf7958e053890089db6a82ad491fb5b9d
        Reviewed-on: http://gerrit.cloudera.org:8080/6642
        Tested-by: Impala Public Jenkins
        Reviewed-by: Matthew Jacobs <mj@cloudera.com>

        Show
        twmarshall Thomas Tauber-Marshall added a comment - commit 1fa331537acf4c2460f3787ba81a5a64cac89c99 Author: Thomas Tauber-Marshall <tmarshall@cloudera.com> Date: Fri Apr 14 13:28:52 2017 -0700 IMPALA-5080 : OutOfMemory PermGen space The problem is that over time our usage of PermGen space has gradually gone up, recently hitting a point where we may run out during testing. The fix is to increase the allowed max perm gen space. The default is 64mb on 32bit systems and 82mb on 64bit systems. After bumping it up to 128mb, I haven't seen any failures on my machine. I also verified that this is not a leak by running the test, test_java_udfs, 10 times, checking the perm gen usage with jmap, running it a further 300 times, and checking that the perm gen usage remained the same. Change-Id: Iccfb69cdf7958e053890089db6a82ad491fb5b9d Reviewed-on: http://gerrit.cloudera.org:8080/6642 Tested-by: Impala Public Jenkins Reviewed-by: Matthew Jacobs <mj@cloudera.com>
        Hide
        joemcdonnell Joe McDonnell added a comment -

        Seen in a recent asf exhaustive release run.

        Show
        joemcdonnell Joe McDonnell added a comment - Seen in a recent asf exhaustive release run.
        Hide
        twmarshall Thomas Tauber-Marshall added a comment -

        commit 920641ffdce1618199ac4d18493151acfc96cda6
        Author: Thomas Tauber-Marshall <tmarshall@cloudera.com>
        Date: Wed May 3 11:46:38 2017 -0700

        IMPALA-5080: OutOfMemory PermGen space

        A previous fix for this increased the MaxPermGen size.
        However, during the review process an extraneous character
        was accidentally inserted that rendered the fix ineffective.

        Testing:

        • Ran the test, test_java_udfs, in a loop for a while.

        Change-Id: If0096384e419c0a059f555d325314e46e7ebfdb8
        Reviewed-on: http://gerrit.cloudera.org:8080/6790
        Reviewed-by: Alex Behm <alex.behm@cloudera.com>
        Reviewed-by: Matthew Jacobs <mj@cloudera.com>
        Tested-by: Impala Public Jenkins

        Show
        twmarshall Thomas Tauber-Marshall added a comment - commit 920641ffdce1618199ac4d18493151acfc96cda6 Author: Thomas Tauber-Marshall <tmarshall@cloudera.com> Date: Wed May 3 11:46:38 2017 -0700 IMPALA-5080 : OutOfMemory PermGen space A previous fix for this increased the MaxPermGen size. However, during the review process an extraneous character was accidentally inserted that rendered the fix ineffective. Testing: Ran the test, test_java_udfs, in a loop for a while. Change-Id: If0096384e419c0a059f555d325314e46e7ebfdb8 Reviewed-on: http://gerrit.cloudera.org:8080/6790 Reviewed-by: Alex Behm <alex.behm@cloudera.com> Reviewed-by: Matthew Jacobs <mj@cloudera.com> Tested-by: Impala Public Jenkins

          People

          • Assignee:
            twmarshall Thomas Tauber-Marshall
            Reporter:
            mikesbrown Michael Brown
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development