[DATAFU-173] Change UDAFS to use Aggregator instead of UserDefinedAggregateFunction

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0

Language

scala

• Assignee: Eyal Allweil
• Reporter: Eyal Allweil
• Created: 03/Oct/23 05:52
• Updated: 14/Jan/24 19:03
• Resolved: 03/Jan/24 15:05

Description

Currently our UDAFs use the UserDefinedAggregateFunction class. There are two drawbacks with this:

1) It is less efficient than Aggregator

2) UserDefinedAggregateFunction is deprecated and removed from Spark 3.2.0.

 

This story is for changing them to use Aggregator.

 

The UDAFs are located here:

https://github.com/apache/datafu/blob/main/datafu-spark/src/main/scala/datafu/spark/SparkUDAFs.scala

 

Here are some links explaining how to do this:

https://stackoverflow.com/questions/48180598/spark-what-is-the-difference-between-aggregator-and-udaf

https://stackoverflow.com/questions/66808917/apache-spark-how-to-define-a-userdefinedaggregatefunction-after-3

 

This change should be backwards compatible if possible; the tests in TestSparkUDAFs should all still pass.