Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-27297

Add higher order functions to Scala API

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0
    • Fix Version/s: 3.0.0
    • Component/s: SQL
    • Labels:
      None

      Description

      There is currently no existing Scala API equivalent for the higher order functions introduced in Spark 2.4.0.

      • transform
      • aggregate
      • filter
      • exists
      • zip_with
      • map_zip_with
      • map_filter
      • transform_values
      • transform_keys

      Equivalent column based functions should be added to the Scala API for org.apache.spark.sql.functions with the following signatures:

       

      def transform(column: Column, f: Column => Column): Column = ???
      
      def transform(column: Column, f: (Column, Column) => Column): Column = ???
      
      def exists(column: Column, f: Column => Column): Column = ???
      
      def filter(column: Column, f: Column => Column): Column = ???
      
      def aggregate(
      expr: Column,
      zero: Column,
      merge: (Column, Column) => Column,
      finish: Column => Column): Column = ???
      
      def aggregate(
      expr: Column,
      zero: Column,
      merge: (Column, Column) => Column): Column = ???
      
      def zip_with(
      left: Column,
      right: Column,
      f: (Column, Column) => Column): Column = ???
      
      def transform_keys(expr: Column, f: (Column, Column) => Column): Column = ???
      
      def transform_values(expr: Column, f: (Column, Column) => Column): Column = ???
      
      def map_filter(expr: Column, f: (Column, Column) => Column): Column = ???
      
      def map_zip_with(left: Column, right: Column, f: (Column, Column, Column) => Column): Column = ???
      
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                nikvanderhoof Nikolas Vanderhoof
                Reporter:
                nikvanderhoof Nikolas Vanderhoof
              • Votes:
                1 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: