[SPARK-21390] Dataset filter api inconsistency

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Incomplete
• Affects Version/s: 2.0.1, 2.1.0, 2.2.0
• Component/s: SQL
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Gheorghe Gheorghe
• Created: 12/Jul/17 15:54
• Updated: 21/May/19 04:15
• Resolved: 21/May/19 04:15

Description

Hello everybody,

I've encountered a strange situation with the spark-shell.
When I run the code below in my IDE the second test case prints as expected count "1". However, when I run the same code using the spark-shell in the second test case I get 0 back as a count.
I've made sure that I'm running scala 2.11.8 and spark 2.0.1 in both my IDE and spark-shell.

  import org.apache.spark.sql.Dataset

  case class SomeClass(field1:String, field2:String)

  val filterCondition: Seq[SomeClass] = Seq( SomeClass("00", "01") )

  // Test 1
  val filterMe1: Dataset[SomeClass] = Seq( SomeClass("00", "01") ).toDS
  
  println("Works fine!" +filterMe1.filter(filterCondition.contains(_)).count)
  
  // Test 2
  case class OtherClass(field1:String, field2:String)
  
  val filterMe2 = Seq( OtherClass("00", "01"), OtherClass("00", "02")).toDS

  println("Fail, count should return 1: " + filterMe2.filter(x=> filterCondition.contains(SomeClass(x.field1, x.field2))).count)

Note if I transform the dataset first I get 1 back as expected.

 println(filterMe2.map(x=> SomeClass(x.field1, x.field2)).filter(filterCondition.contains(_)).count)

Is this a bug? I can see that this filter function has been marked as experimental https://spark.apache.org/docs/2.1.0/api/java/org/apache/spark/sql/Dataset.html#filter(scala.Function1)