[HIVE-18690] Integrate with Spark OutputMetrics

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0-alpha-1
• Component/s: Spark
• Target Version/s: 3.1.0
• Assignee: Sahil Takiar
• Reporter: Sahil Takiar
• Created: 13/Feb/18 03:11
• Updated: 17/Nov/22 08:55
• Resolved: 07/Jun/18 19:31

Description

Spark has an OutputMetrics it uses to expose records / bytes written. We currently don't integrate with it and the Spark UI shows a blank value for output records / bytes. We have our own customer accumulators instead (like HIVE_RECORDS_OUT).

Spark exposes the OutputMetrics object inside individual tasks via the TaskContext.get() method. We can use this method to access the OutputMetrics object and update it.