Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-37061

Custom V2 Metrics uses wrong classname for lookup

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 3.2.0
    • 3.2.1, 3.3.0
    • SQL
    • None

    Description

      Currently CustomMetrics uses `getCanonicalName` to get the metric type name

      https://github.com/apache/spark/blob/38493401d18d42a6cb176bf515536af97ba1338b/sql/core/src/main/scala/org/apache/spark/sql/execution/metric/CustomMetrics.scala#L31-L33

      But when using reflection we need to use the original type name.

      Here is an example when working with an inner class

      Class.getName = org.apache.iceberg.spark.source.SparkBatchScan$FilesScannedMetric
      Class.getCanonicalName =
      org.apache.iceberg.spark.source.SparkBatchScan.FilesScannedMetric
      

      The "$" name is required to look up this class while the "." version will fail with CNF.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            rspitzer Russell Spitzer
            rspitzer Russell Spitzer
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment