[SPARK-37578] DSV2 is not updating Output Metrics

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.3.0
• Component/s: Spark Core
• Assignee: L. C. Hsieh
• Reporter: Sandeep Katta
• Created: 08/Dec/21 09:28
• Updated: 22/Jan/22 07:35
• Resolved: 29/Dec/21 06:09

Description

Repro code

./bin/spark-shell --master local  --jars /Users/jars/iceberg-spark3-runtime-0.12.1.jar

 

import scala.collection.mutable
import org.apache.spark.scheduler._val bytesWritten = new mutable.ArrayBuffer[Long]()
val recordsWritten = new mutable.ArrayBuffer[Long]()
val bytesWrittenListener = new SparkListener() {
  override def onTaskEnd(taskEnd: SparkListenerTaskEnd): Unit = {
    bytesWritten += taskEnd.taskMetrics.outputMetrics.bytesWritten
    recordsWritten += taskEnd.taskMetrics.outputMetrics.recordsWritten
  }
}
spark.sparkContext.addSparkListener(bytesWrittenListener)
try {
val df = spark.range(1000).toDF("id")
  df.write.format("iceberg").save("Users/data/dsv2_test")
  
assert(bytesWritten.sum > 0)
assert(recordsWritten.sum > 0)
} finally {
  spark.sparkContext.removeSparkListener(bytesWrittenListener)
} 

 

 

Issue Links

• links to: [Github] Pull Request #35028 (viirya)
• links to: [Github] Pull Request #35277 (AngersZhuuuu)