[SPARK-34338] Report metrics from Datasource v2 scan

• Type: Umbrella
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.2.0
• Component/s: SQL
• Assignee: L. C. Hsieh
• Reporter: L. C. Hsieh
• Created: 03/Feb/21 04:14
• Updated: 20/Apr/21 15:02
• Resolved: 20/Apr/21 15:02

Description

This is related to SPARK-34297.

In SPARK-34297, we want to add a couple of useful metrics when reading from Kafka in SS. We need some public API change in DS v2 to make it possible.

Issue Links

• links to: [Github] Pull Request #31451 (viirya)

Sub-Tasks

• Add metric interfaces to DS v2: Resolved
• Batch implementation for metrics from Datasource v2 scan: Open
• Streaming implementation for metrics from Datasource v2 scan: Open