Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-15703

Make ListenerBus event queue size configurable

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.0.0
    • Fix Version/s: 2.0.1, 2.1.0
    • Component/s: Scheduler, Web UI
    • Labels:
      None
    • Target Version/s:

      Description

      The Spark UI doesn't seem to be showing all the tasks and metrics.

      I ran a job with 100000 tasks but Detail stage page says it completed 93029:

      Summary Metrics for 93029 Completed Tasks

      The Stages for all jobs pages list that only 89519/100000 tasks finished but its completed. The metrics for shuffled write and input are also incorrect.

      I will attach screen shots.

      I checked the logs and it does show that all the tasks actually finished.

      16/06/01 16:15:42 INFO TaskSetManager: Finished task 59880.0 in stage 2.0 (TID 54038) in 265309 ms on 10.213.45.51 (100000/100000)
      16/06/01 16:15:42 INFO YarnClusterScheduler: Removed TaskSet 2.0, whose tasks have all completed, from pool

        Attachments

        1. SparkListenerBus .png
          76 kB
          Dhruve Ashar
        2. spark-dynamic-executor-allocation.png
          179 kB
          Dhruve Ashar
        3. Screen Shot 2016-06-01 at 11.23.48 AM.png
          411 kB
          Thomas Graves
        4. Screen Shot 2016-06-01 at 11.21.32 AM.png
          481 kB
          Thomas Graves

          Issue Links

            Activity

              People

              • Assignee:
                Dhruve Ashar Dhruve Ashar
                Reporter:
                tgraves Thomas Graves
              • Votes:
                0 Vote for this issue
                Watchers:
                12 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: