Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.4.0
    • Fix Version/s: 1.4.0
    • Component/s: Web UI
    • Labels:
      None
    • Target Version/s:

      Description

      I sometimes trouble-shoot and analyse the cause of long time spending job.

      At the time, I find the stages which spends long time or fails, then I find the tasks which spends long time or fails, next I analyse the proportion of each phase in a task.

      Another case, I find executors which spends long time for running a task and analyse the details of a task.

      In such situation, I think it's helpful to visualize timeline view of stages / tasks / executors and visualize details of proportion of activity for each task.

      Now I'm developing prototypes like captures I attached.
      I'll integrate these viewer into WebUI.

      1. ApplicationTimeliView.png
        233 kB
        Kousuke Saruta
      2. JobTimelineView.png
        182 kB
        Kousuke Saruta
      3. TaskAssignmentTimelineView.png
        299 kB
        Kousuke Saruta

        Issue Links

          Activity

          Hide
          pwendell Patrick Wendell added a comment - - edited

          Thanks for the contribution Kousuke Saruta, I've merged this.

          Show
          pwendell Patrick Wendell added a comment - - edited Thanks for the contribution Kousuke Saruta , I've merged this.
          Hide
          sarutak Kousuke Saruta added a comment -

          I've attached the screen shots of this feature.

          Show
          sarutak Kousuke Saruta added a comment - I've attached the screen shots of this feature.
          Hide
          sarutak Kousuke Saruta added a comment -

          I added 3 timeline features to existing Web UI.

          [Application Timeline View]
          This view shows following things.

          When each executor was added/removed and the reason why it's removed.
          When each job was started/finished.
          Status of each job.

          [Stage Timeline View]
          Similar to Application Timeline View, this view shows following things.

          When each executor was added/removed and the reason why it's removed.
          When each job was started/finished.
          Status of each stage.

          [Task Assignment Timeline View]
          This view shows following things.

          When each task started/finished
          How long each task spent and the proportion.
          Status of each task.
          Where each task ran on.

          All the view above is zoomable by mouse wheel action and scrollable by drag action.

          Show
          sarutak Kousuke Saruta added a comment - I added 3 timeline features to existing Web UI. [Application Timeline View] This view shows following things. When each executor was added/removed and the reason why it's removed. When each job was started/finished. Status of each job. [Stage Timeline View] Similar to Application Timeline View, this view shows following things. When each executor was added/removed and the reason why it's removed. When each job was started/finished. Status of each stage. [Task Assignment Timeline View] This view shows following things. When each task started/finished How long each task spent and the proportion. Status of each task. Where each task ran on. All the view above is zoomable by mouse wheel action and scrollable by drag action.
          Hide
          sandyr Sandy Ryza added a comment -

          This looks like a really cool addition.

          Show
          sandyr Sandy Ryza added a comment - This looks like a really cool addition.
          Hide
          apachespark Apache Spark added a comment -

          User 'sarutak' has created a pull request for this issue:
          https://github.com/apache/spark/pull/2342

          Show
          apachespark Apache Spark added a comment - User 'sarutak' has created a pull request for this issue: https://github.com/apache/spark/pull/2342

            People

            • Assignee:
              sarutak Kousuke Saruta
              Reporter:
              sarutak Kousuke Saruta
            • Votes:
              1 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development