The Spark UI doesn't seem to be showing all the tasks and metrics.
I ran a job with 100000 tasks but Detail stage page says it completed 93029:
Summary Metrics for 93029 Completed Tasks
The Stages for all jobs pages list that only 89519/100000 tasks finished but its completed. The metrics for shuffled write and input are also incorrect.
I will attach screen shots.
I checked the logs and it does show that all the tasks actually finished.
16/06/01 16:15:42 INFO TaskSetManager: Finished task 59880.0 in stage 2.0 (TID 54038) in 265309 ms on 10.213.45.51 (100000/100000)
16/06/01 16:15:42 INFO YarnClusterScheduler: Removed TaskSet 2.0, whose tasks have all completed, from pool