Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-24104

SQLAppStatusListener overwrites metrics onDriverAccumUpdates instead of updating them

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.3.0
    • 2.3.1, 2.4.0
    • SQL
    • None

    Description

      SqlAppStatusListener does

      exec.driverAccumUpdates = accumUpdates.toMap
      update(exec)
      

      in onDriverAccumUpdates.
      But postDriverMetricUpdates is called multiple time per query, e.g. from each FileSourceScanExec and BroadcastExchangeExec.

      If the update does not really update it in the KV store (depending on liveUpdatePeriodNs), the previously posted metrics are lost.

      Attachments

        Activity

          People

            juliuszsompolski Juliusz Sompolski
            juliuszsompolski Juliusz Sompolski
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: