Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-22510 Exceptions caused by 64KB JVM bytecode or 64K constant pool entry limit
  3. SPARK-22226

splitExpression can create too many method calls (generating a Constant Pool limit error)

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.2.0
    • Fix Version/s: 2.3.0
    • Component/s: SQL
    • Labels:
      None

      Description

      Code generation for very wide datasets can fail because of the Constant Pool limit reached.

      This can be caused by many reasons. One of them is that we are currently splitting the definition of the generated methods among several NestedClass but all these methods are called in the main class. Since we have entries added to the constant pool for each method invocation, this is limiting the number of rows and is leading for very wide dataset to:

      org.codehaus.janino.JaninoRuntimeException: Constant pool for class org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificMutableProjection has grown past JVM limit of 0xFFFF
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                mgaido Marco Gaido
                Reporter:
                mgaido Marco Gaido
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: