[SPARK-15153] SparkR spark.naiveBayes throws error when label is numeric type

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.1.0
• Component/s: ML, SparkR
• Target Version/s: 2.1.0
• Assignee: Yanbo Liang
• Reporter: Yanbo Liang
• Shepherd: Xiangrui Meng
• Created: 05/May/16 10:32
• Updated: 11/Oct/16 19:41
• Resolved: 11/Oct/16 19:41

Description

When the label of dataset is numeric type, SparkR spark.naiveBayes will throw error. This bug is easy to reproduce:

t <- as.data.frame(Titanic)
t1 <- t[t$Freq > 0, -5]
t1$NumericSurvived <- ifelse(t1$Survived == "No", 0, 1)
t2 <- t1[-4]
df <- suppressWarnings(createDataFrame(sqlContext, t2))
m <- spark.naiveBayes(df, NumericSurvived ~ .)

16/05/05 03:26:17 ERROR RBackendHandler: fit on org.apache.spark.ml.r.NaiveBayesWrapper failed
Error in invokeJava(isStatic = TRUE, className, methodName, ...) :
  java.lang.ClassCastException: org.apache.spark.ml.attribute.UnresolvedAttribute$ cannot be cast to org.apache.spark.ml.attribute.NominalAttribute
	at org.apache.spark.ml.r.NaiveBayesWrapper$.fit(NaiveBayesWrapper.scala:66)
	at org.apache.spark.ml.r.NaiveBayesWrapper.fit(NaiveBayesWrapper.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.api.r.RBackendHandler.handleMethodCall(RBackendHandler.scala:141)
	at org.apache.spark.api.r.RBackendHandler.channelRead0(RBackendHandler.scala:86)
	at org.apache.spark.api.r.RBackendHandler.channelRead0(RBackendHandler.scala:38)
	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
	at io.netty.channel.AbstractChannelHandlerContext.invo

In RFormula, the response variable type could be string or numeric. If it's string, RFormula will transform it to label of DoubleType by StringIndexer and set corresponding column metadata; otherwise, RFormula will directly use it as label when training model (and assumes that it was numbered from 0, ..., maxLabelIndex).
When we extract labels at ml.r.NaiveBayesWrapper, we should handle it according the type of the response variable (string or numeric).

cc mengxr josephkb

Issue Links

• depends upon: SPARK-15957 RFormula supports forcing to index label
• Is contained by: SPARK-15540 RFormula and R feature processing improvement umbrella
• is duplicated by: SPARK-15510 SparkR NaiveBayes should not require label to have NominalAttribute
• relates to: SPARK-11107 spark.ml should support more input column types: umbrella
• links to: [Github] Pull Request #12930 (yanboliang)
• links to: [Github] Pull Request #15431 (yanboliang)