Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Invalid
-
2.2.1, 2.3.0
-
None
-
None
Description
The following code can throw an exception with or without whole-stage codegen.
public void SPARK22935() { Dataset<CDR> cdr = spark .read() .format("csv") .option("header", "true") .option("inferSchema", "true") .option("delimiter", ";") .csv("CDR_SAMPLE.csv") .as(Encoders.bean(CDR.class)); Dataset<CDR> ds = cdr.filter((FilterFunction<CDR>) x -> (x.timestamp != null)); long c = ds.count(); cdr.show(2); ds.show(2); System.out.println("cnt=" + c); } // CDR.java public class CDR implements java.io.Serializable { public java.sql.Date timestamp; public java.sql.Date getTimestamp() { return this.timestamp; } public void setTimestamp(java.sql.Date timestamp) { this.timestamp = timestamp; } } // CDR_SAMPLE.csv timestamp 2017-10-29T02:37:07.815Z 2017-10-29T02:38:07.815Z
result
12:17:10.352 ERROR org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 61, Column 70: No applicable constructor/method found for actual parameters "long"; candidates are: "public static java.sql.Date org.apache.spark.sql.catalyst.util.DateTimeUtils.toJavaDate(int)" org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 61, Column 70: No applicable constructor/method found for actual parameters "long"; candidates are: "public static java.sql.Date org.apache.spark.sql.catalyst.util.DateTimeUtils.toJavaDate(int)" at org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:11821) ...