I have an application in which a lot of if-else decisioning is involved to generate output. I'm getting following exception:
Caused by: org.codehaus.janino.JaninoRuntimeException: Code of method "(Lorg/apache/spark/sql/catalyst/expressions/GeneratedClass$SpecificUnsafeProjection;Lorg/apache/spark/sql/catalyst/InternalRow;)V" of class "org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection" grows beyond 64 KB
Steps to Reproduce:
I've come up with a unit test which I was able to run in CodeGenerationSuite.scala:
Current splitting of Projection codes doesn't (and can't) take care of splitting the generated code for individual output column expressions. So it can grow to exceed JVM limit.
Note: This issue seems related to
SPARK-14887 but I'm not sure whether the root cause is same
If expression should place it's predicate, true value and false value expressions' generated code in separate methods in context and call these methods instead of putting the whole code directly in its generated code