Description
There is currently no existing Scala API equivalent for the higher order functions introduced in Spark 2.4.0.
- transform
- aggregate
- filter
- exists
- zip_with
- map_zip_with
- map_filter
- transform_values
- transform_keys
Equivalent column based functions should be added to the Scala API for org.apache.spark.sql.functions with the following signatures:
def transform(column: Column, f: Column => Column): Column = ??? def transform(column: Column, f: (Column, Column) => Column): Column = ??? def exists(column: Column, f: Column => Column): Column = ??? def filter(column: Column, f: Column => Column): Column = ??? def aggregate( expr: Column, zero: Column, merge: (Column, Column) => Column, finish: Column => Column): Column = ??? def aggregate( expr: Column, zero: Column, merge: (Column, Column) => Column): Column = ??? def zip_with( left: Column, right: Column, f: (Column, Column) => Column): Column = ??? def transform_keys(expr: Column, f: (Column, Column) => Column): Column = ??? def transform_values(expr: Column, f: (Column, Column) => Column): Column = ??? def map_filter(expr: Column, f: (Column, Column) => Column): Column = ??? def map_zip_with(left: Column, right: Column, f: (Column, Column, Column) => Column): Column = ???
Attachments
Issue Links
- relates to
-
SPARK-19480 Higher order functions in SQL
-
- Resolved
-
-
SPARK-30681 Add higher order functions API to PySpark
-
- Resolved
-
-
-
- Resolved
-
- links to