[SPARK-36319] Have Observation return Map instead of Row

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.3.0
• Component/s: Java API, PySpark, SQL
• Assignee: Enrico Minack
• Reporter: Enrico Minack
• Shepherd: Hyukjin Kwon
• Created: 27/Jul/21 19:37
• Updated: 12/Dec/22 18:10
• Resolved: 01/Aug/21 01:41

Description

As gurwls223 pointed out, the Observation API (Scala, Java, PySpark) could return a Map / Dict. It currently returns Row simply because the metrics are (internal to Observation) retrieved from the listener as rows. Since that is hidden from the user by the Observation API, there is no need to return Row.

If there is some value in the original Row, both could be provided via getAsRow and getAsMap.

The Observation API has been added to Spark in unreleased 3.3.0, so it should not be a blocker to remove the Row return type in 3.3.0 again.

Issue Links

• links to: [Github] Pull Request #33545 (EnricoMi)
• links to: [Github] Pull Request #33545 (EnricoMi)