Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-27297

Add higher order functions to Scala API

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.0.0
    • 3.0.0
    • SQL
    • None

    Description

      There is currently no existing Scala API equivalent for the higher order functions introduced in Spark 2.4.0.

      • transform
      • aggregate
      • filter
      • exists
      • zip_with
      • map_zip_with
      • map_filter
      • transform_values
      • transform_keys

      Equivalent column based functions should be added to the Scala API for org.apache.spark.sql.functions with the following signatures:

       

      def transform(column: Column, f: Column => Column): Column = ???
      
      def transform(column: Column, f: (Column, Column) => Column): Column = ???
      
      def exists(column: Column, f: Column => Column): Column = ???
      
      def filter(column: Column, f: Column => Column): Column = ???
      
      def aggregate(
      expr: Column,
      zero: Column,
      merge: (Column, Column) => Column,
      finish: Column => Column): Column = ???
      
      def aggregate(
      expr: Column,
      zero: Column,
      merge: (Column, Column) => Column): Column = ???
      
      def zip_with(
      left: Column,
      right: Column,
      f: (Column, Column) => Column): Column = ???
      
      def transform_keys(expr: Column, f: (Column, Column) => Column): Column = ???
      
      def transform_values(expr: Column, f: (Column, Column) => Column): Column = ???
      
      def map_filter(expr: Column, f: (Column, Column) => Column): Column = ???
      
      def map_zip_with(left: Column, right: Column, f: (Column, Column, Column) => Column): Column = ???
      
      

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            nikvanderhoof Nikolas Vanderhoof
            nikvanderhoof Nikolas Vanderhoof
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Issue deployment