[SPARK-15285] Generated SpecificSafeProjection.apply method grows beyond 64 KB

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.6.1, 2.0.0
• Fix Version/s: 2.0.1, 2.1.0
• Component/s: SQL
• Assignee: Kazuaki Ishizaki
• Reporter: Konstantin Shaposhnikov
• Created: 12/May/16 02:15
• Updated: 14/Jan/17 20:05
• Resolved: 22/Aug/16 07:49

Description

The following code snippet results in

 org.codehaus.janino.JaninoRuntimeException: Code of method "(Ljava/lang/Object;)Ljava/lang/Object;" of class "org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificSafeProjection" grows beyond 64 KB
  at org.codehaus.janino.CodeContext.makeSpace(CodeContext.java:941)

case class S100(s1:String="1", s2:String="2", s3:String="3", s4:String="4", s5:String="5", s6:String="6", s7:String="7", s8:String="8", s9:String="9", s10:String="10", s11:String="11", s12:String="12", s13:String="13", s14:String="14", s15:String="15", s16:String="16", s17:String="17", s18:String="18", s19:String="19", s20:String="20", s21:String="21", s22:String="22", s23:String="23", s24:String="24", s25:String="25", s26:String="26", s27:String="27", s28:String="28", s29:String="29", s30:String="30", s31:String="31", s32:String="32", s33:String="33", s34:String="34", s35:String="35", s36:String="36", s37:String="37", s38:String="38", s39:String="39", s40:String="40", s41:String="41", s42:String="42", s43:String="43", s44:String="44", s45:String="45", s46:String="46", s47:String="47", s48:String="48", s49:String="49", s50:String="50", s51:String="51", s52:String="52", s53:String="53", s54:String="54", s55:String="55", s56:String="56", s57:String="57", s58:String="58", s59:String="59", s60:String="60", s61:String="61", s62:String="62", s63:String="63", s64:String="64", s65:String="65", s66:String="66", s67:String="67", s68:String="68", s69:String="69", s70:String="70", s71:String="71", s72:String="72", s73:String="73", s74:String="74", s75:String="75", s76:String="76", s77:String="77", s78:String="78", s79:String="79", s80:String="80", s81:String="81", s82:String="82", s83:String="83", s84:String="84", s85:String="85", s86:String="86", s87:String="87", s88:String="88", s89:String="89", s90:String="90", s91:String="91", s92:String="92", s93:String="93", s94:String="94", s95:String="95", s96:String="96", s97:String="97", s98:String="98", s99:String="99", s100:String="100")

case class S(s1: S100=S100(), s2: S100=S100(), s3: S100=S100(), s4: S100=S100(), s5: S100=S100(), s6: S100=S100(), s7: S100=S100(), s8: S100=S100(), s9: S100=S100(), s10: S100=S100())

val ds = Seq(S(),S(),S()).toDS
ds.show()

I could reproduce this with Spark built from 1.6 branch and with https://home.apache.org/~pwendell/spark-nightly/spark-master-bin/spark-2.0.0-SNAPSHOT-2016_05_11_01_03-8beae59-bin/

Issue Links

• is duplicated by: SPARK-18492 GeneratedIterator grows beyond 64 KB
• relates to: SPARK-18118 SpecificSafeProjection.apply of Java Object from Dataset to JavaRDD Grows Beyond 64 KB
• links to: [Github] Pull Request #13243 (kiszk)
• links to: [Github] Pull Request #14670 (kiszk)