[SPARK-30306] Instrument Python UDF execution time and metrics using Spark Metrics system

• Type: Improvement
• Status: In Progress
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 3.1.0
• Component/s: PySpark, Spark Core
• Assignee: Unassigned
• Reporter: Luca Canali
• Created: 19/Dec/19 14:54
• Updated: 30/Apr/20 16:48

Description

This proposes to extend Spark instrumentation to add metrics aimed at understanding the performance of Python code called by Spark, via UDF, Pandas UDF or with MapPartittions. Relevant performance counters are exposed using the Spark Metrics System (based on the Dropwizard library).  This allows to easily consume the metrics produced by executors, for example using a performance dashboard. See also the attached screenshot.

Issue Links

• links to: GitHub Pull Request #26953