Major Description
Implemented isInCollection in DataFrame API for both Scala and Java, so users can do
val profileDF = Seq( Some(1), Some(2), Some(3), Some(4), Some(5), Some(6), Some(7), None ).toDF("profileID") val validUsers: Set[Any] = Set(6, 7.toShort, 8L, "3") val result = profileDF.withColumn("isValid", $"profileID".isInCollection(validUsers)) result.show(10) """ +----------+------+ |profileID|isValid| +----------+------+ |1|false| |2|false| |3|true| |4|false| |5|false| |6|true| |7|true| |null|null| +----------+------+ """.stripMargin
