[SPARK-46663] Disable memory profiler for pandas UDFs with iterators

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.5.0, 4.0.0
• Fix Version/s: 3.5.1
• Component/s: PySpark
• Labels: pull-request-available
• Assignee: Xinrong Meng
• Reporter: Xinrong Meng
• Created: 10/Jan/24 22:12
• Updated: 18/Jan/24 00:01
• Resolved: 18/Jan/24 00:01

Description

When using pandas UDFs with iterators, if users enable the profiling spark conf, a warning indicating non-support should be raised, and profiling should be disabled.

However, currently, after raising the not-supported warning, the memory profiler is still being enabled.

We should fix that.

Issue Links

• links to: GitHub Pull Request #44668
• links to: GitHub Pull Request #44760