Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-36319

Have Observation return Map instead of Row

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.3.0
    • Fix Version/s: 3.3.0
    • Component/s: Java API, PySpark, SQL
    • Labels:
      None

      Description

      As Hyukjin Kwon pointed out, the Observation API (Scala, Java, PySpark) could return a Map / Dict. It currently returns Row simply because the metrics are (internal to Observation) retrieved from the listener as rows. Since that is hidden from the user by the Observation API, there is no need to return Row.

      If there is some value in the original Row, both could be provided via getAsRow and getAsMap.

      The Observation API has been added to Spark in unreleased 3.3.0, so it should not be a blocker to remove the Row return type in 3.3.0 again.

        Attachments

          Activity

            People

            • Assignee:
              EnricoMi Enrico Minack
              Reporter:
              EnricoMi Enrico Minack
              Shepherd:
              Hyukjin Kwon
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: