Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-3983

Scheduler delay (shown in the UI) is incorrect

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.2.0
    • Component/s: Web UI
    • Labels:
      None

      Description

      The reported scheduler delay includes time to get a new thread (from a threadpool) in order to start the task, time to deserialize the task, and time to serialize the result. None of these things are delay caused by the scheduler; including them as such is misleading.

      This is especially problematic when debugging performance of short tasks (that run in 10s of milliseconds), when the scheduler delay can be very large relative to the task duration.

      cc Evan Sparks Shivaram Venkataraman

        Attachments

          Activity

            People

            • Assignee:
              kayousterhout Kay Ousterhout
              Reporter:
              kayousterhout Kay Ousterhout
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: