[SPARK-16845] org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificOrdering" grows beyond 64 KB

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 1.6.4, 2.0.3, 2.1.1, 2.2.0
• Component/s: SQL
• Flags: Important
• Assignee: Liwei Lin
• Reporter: hejie
• Created: 02/Aug/16 02:04
• Updated: 07/Feb/18 01:40
• Resolved: 10/Jan/17 11:37

Description

I have a wide table(400 columns), when I try fitting the traindata on all columns, the fatal error occurs.

... 46 more
Caused by: org.codehaus.janino.JaninoRuntimeException: Code of method "(Lorg/apache/spark/sql/catalyst/InternalRow;Lorg/apache/spark/sql/catalyst/InternalRow;)I" of class "org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificOrdering" grows beyond 64 KB
at org.codehaus.janino.CodeContext.makeSpace(CodeContext.java:941)
at org.codehaus.janino.CodeContext.write(CodeContext.java:854)

Issue Links

• Blocked: SPARK-21828 org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificOrdering" grows beyond 64 KB...again
• is duplicated by: SPARK-17092 DataFrame with large number of columns causing code generation error
• is duplicated by: SPARK-17131 Code generation fails when running SQL expressions against a wide dataset (thousands of columns)
• is duplicated by: SPARK-17223 "grows beyond 64 KB" with data frame with many columns
• is duplicated by: SPARK-18492 GeneratedIterator grows beyond 64 KB
• is related to: SPARK-18207 class "org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection" grows beyond 64 KB
• links to: [Github] Pull Request #15461 (lw-lin)
• links to: [Github] Pull Request #15480 (lw-lin)
• links to: [Github] Pull Request #17157 (ueshin)
• links to: [Github] Pull Request #17158 (ueshin)