Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-3983

Scheduler delay (shown in the UI) is incorrect

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 1.2.0
    • Web UI
    • None

    Description

      The reported scheduler delay includes time to get a new thread (from a threadpool) in order to start the task, time to deserialize the task, and time to serialize the result. None of these things are delay caused by the scheduler; including them as such is misleading.

      This is especially problematic when debugging performance of short tasks (that run in 10s of milliseconds), when the scheduler delay can be very large relative to the task duration.

      cc sparks shivaram

      Attachments

        Activity

          People

            kayousterhout Kay Ousterhout
            kayousterhout Kay Ousterhout
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: