[SPARK-17728] UDFs are run too many times

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Not A Problem
• Affects Version/s: 2.0.0
• Component/s: Spark Core

Environment

Databricks Cloud / Spark 2.0.0

• Assignee: Unassigned
• Reporter: Jacob Eisinger
• Created: 29/Sep/16 17:43
• Updated: 23/Jun/22 01:09
• Resolved: 30/Sep/16 02:27

Description

Background

Llonger running processes that might run analytics or contact external services from UDFs. The response might not just be a field, but instead a structure of information. When attempting to break out this information, it is critical that query is optimized correctly.

Steps to Reproduce

1. Create some sample data.
2. Create a UDF that returns a multiple attributes.
3. Run UDF over some data.
4. Create new columns from the multiple attributes.
5. Observe run time.

Actual Results

The UDF is executed multiple times per row.

Expected Results

The UDF should only be executed once per row.

Workaround

Cache the Dataset after UDF execution.

Details

For code and more details, see over_optimized_udf.html

Issue Links

• Blocked: SPARK-27245 Optimizer repeat Python UDF calls
• duplicates: SPARK-12352 Reuse the result of split in SQL
• is duplicated by: SPARK-18748 UDF multiple evaluations causes very poor performance
• is duplicated by: SPARK-18747 UDF multiple evaluations causes very poor performance