Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-6702

Update the Streaming Tab in Spark UI to show more batch information

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete CommentsDelete
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • 1.3.1
    • 1.4.0
    • DStreams, Web UI
    • None

    Description

      Since the Streaming Tab was added to the Spark UI in Spark 1.0, its has proven to be a defacto way to debug Spark Streaming applications at the time of development. However, there is much more we can do to help developers debug their streaming applications. Specifically we would like to show the details of individual batches and link them to the jobs that ran for that batch (similar to job details showing stages in the job). Furthermore we would also like show timelines to see trends over time for processing times data rates etc.

      This is the design doc that shows the proposed new Streaming tab in the Spark UI.
      https://docs.google.com/document/d/1-ZjvQ_2thWEQkTxRMHrVdnEI57XTi3wZEBUoqrrDg5c/edit?usp=sharing

      Attachments

        Issue Links

        There are no Sub-Tasks for this issue.

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            zsxwing Shixiong Zhu Assign to me
            tdas Tathagata Das
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment