Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-43334

Error while serializing ExecutorPeakMetricsDistributions into API JSON response

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.3.3
    • 3.5.0
    • Web UI
    • None

    Description

      When we try to get the ExecutorPeakMetricsDistributions through the API (/stages), there is a possibility of encountering an issue while serializing the StagesData into a JSON if the executor metrics are empty. 

       

      The following error is thrown : 

      Caused by: com.fasterxml.jackson.databind.JsonMappingException: -1 (through reference chain: scala.c
      ollection.immutable.$colon$colon[0]->org.apache.spark.status.api.v1.StageData["executorMetricsDistri
      butions"]->org.apache.spark.status.api.v1.ExecutorMetricsDistributions["peakMemoryMetrics"])
              at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.jav
      a:390)
              at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.jav
      a:349) 

      This happens because the indices for the quartiles are populated incorrectly as -1 since the metrics itself are empty and this leads to this exception being thrown.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            tejdeepg Thejdeep
            tejdeepg Thejdeep
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment