Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33132

The 'Shuffle Read Size / Records' field in Stage Summary metrics was shown as 'NaN Undefined'

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 3.0.1
    • Fix Version/s: 3.1.0
    • Component/s: Spark Core
    • Labels:
      None

      Description

      Spark Version: 3.0.1

      Description: The 'Shuffle Read Size / Records' field in Stage Summary metrics was shown as 'NaN Undefind'  when the readBytes value is negative,  as the attachment shows.

      curl http:/hadoop001:18081/api/v1/applications/application_1601774913550_0225/stages/2/0/taskSummary

      {
      "quantiles" : [ 0.05, 0.25, 0.5, 0.75, 0.95 ],
      "executorDeserializeTime" : [ 7.0, 357.0, 390.0, 484.0, 492.0 ],
       ...
      "shuffleReadMetrics" :

      { "readBytes" : [ -2.0, 1775984.0, 1779759.0, 1781727.0, 1788426.0 ], "readRecords" : [ 2001.0, 2002.0, 2002.0, 2002.0, 2002.0 ],  ... }

        Attachments

          Activity

            People

            • Assignee:
              echohlne akiyamaneko
              Reporter:
              echohlne akiyamaneko
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: