[SPARK-39635] Custom driver metrics for Datasource v2

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.4.0
• Component/s: SQL
• Assignee: Karuppayya
• Reporter: Karuppayya
• Created: 29/Jun/22 21:03
• Updated: 12/Aug/22 04:56
• Resolved: 12/Aug/22 04:56

Description

org.apache.spark.sql.connector.metric.CustomMetric captures executor metrics.

There is no way to expose driver side metrics on V2 datasources.

 

Issue Links

• links to: [Github] Pull Request #37205 (karuppayya)
• links to: [Github] Pull Request #37205 (karuppayya)