Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-32135

Show Spark Driver name on Spark history web page

Attach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Wish
    • Status: In Progress
    • Minor
    • Resolution: Unresolved
    • 2.4.5
    • None
    • Spark Core
    • None

    Description

      Our service dynamically creates short-lived YARN clusters in cloud. Spark applications run on these dynamically created clusters. Log data for these applications is stored on a remote file-system. We configure a static instance of SparkHistoryServer to view information on jobs that ran on these clusters.

      Since we are using a single History Server for multiple clusters, it will be useful for users to have information on where the jobs were executed. We would like to display this information on the main web-page, instead of having users go through multiple links to retrieve this information.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            gaurangi Gaurangi Saxena

            Dates

              Created:
              Updated:

              Slack

                Issue deployment