[SPARK-37061] Custom V2 Metrics uses wrong classname for lookup

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.2.1, 3.3.0
• Component/s: SQL
• Assignee: Russell Spitzer
• Reporter: Russell Spitzer
• Created: 19/Oct/21 19:39
• Updated: 21/Oct/21 00:59
• Resolved: 21/Oct/21 00:59

Description

Currently CustomMetrics uses `getCanonicalName` to get the metric type name

https://github.com/apache/spark/blob/38493401d18d42a6cb176bf515536af97ba1338b/sql/core/src/main/scala/org/apache/spark/sql/execution/metric/CustomMetrics.scala#L31-L33

But when using reflection we need to use the original type name.

Here is an example when working with an inner class

Class.getName = org.apache.iceberg.spark.source.SparkBatchScan$FilesScannedMetric
Class.getCanonicalName =
org.apache.iceberg.spark.source.SparkBatchScan.FilesScannedMetric

The "$" name is required to look up this class while the "." version will fail with CNF.

Issue Links

• links to: [Github] Pull Request #34344 (RussellSpitzer)
• links to: [Github] Pull Request #34345 (RussellSpitzer)