Description
I have met a problem like https://issues.apache.org/jira/browse/SPARK-3582,
but not with this parameter Map, my evaluate function return a Map:
public Map<String, String> evaluate(Text url)
when run spark-sql with this udf, getting the following exception:
scala.MatchError: interface java.util.Map (of class java.lang.Class)
at org.apache.spark.sql.hive.HiveInspectors$class.javaClassToDataType(HiveInspectors.scala:175)
at org.apache.spark.sql.hive.HiveSimpleUdf.javaClassToDataType(hiveUdfs.scala:112)
at org.apache.spark.sql.hive.HiveSimpleUdf.dataType$lzycompute(hiveUdfs.scala:144)
at org.apache.spark.sql.hive.HiveSimpleUdf.dataType(hiveUdfs.scala:144)
at org.apache.spark.sql.catalyst.expressions.Alias.toAttribute(namedExpressions.scala:133)
at org.apache.spark.sql.catalyst.plans.logical.Project$$anonfun$output$1.apply(basicOperators.scala:25)
at org.apache.spark.sql.catalyst.plans.logical.Project$$anonfun$output$1.apply(basicOperators.scala:25)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.AbstractTraversable.map(Traversable.scala:105)
at org.apache.spark.sql.catalyst.plans.logical.Project.output(basicOperators.scala:25)
at org.apache.spark.sql.catalyst.plans.logical.InsertIntoTable.resolved$lzycompute(basicOperators.scala:149)
Attachments
Issue Links
- is related to
-
SPARK-6912 Throw an AnalysisException when unsupported Java Map<K,V> types used in Hive UDF
-
- Resolved
-