Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-24371

Added isInCollection in DataFrame API for Scala and Java.

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.3.0
    • Fix Version/s: 2.4.0
    • Component/s: SQL
    • Labels:
      None

      Description

      Implemented isInCollection in DataFrame API for both Scala and Java, so users can do

       val profileDF = Seq(
       Some(1), Some(2), Some(3), Some(4),
       Some(5), Some(6), Some(7), None
       ).toDF("profileID")
      
      val validUsers: Set[Any] = Set(6, 7.toShort, 8L, "3")
      
      val result = profileDF.withColumn("isValid", $"profileID".isInCollection(validUsers))
      
      result.show(10)
       """
       +----------+------+
      |profileID|isValid|
      
      +----------+------+
      |1|false|
      |2|false|
      |3|true|
      |4|false|
      |5|false|
      |6|true|
      |7|true|
      |null|null|
      
      +----------+------+
       """.stripMargin
      

        Attachments

          Activity

            People

            • Assignee:
              dbtsai DB Tsai
              Reporter:
              dbtsai DB Tsai
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: