[SPARK-34877] Add Spark AM Log link in case of master as yarn and deploy mode as client

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.1.1
• Fix Version/s: 3.2.0
• Component/s: Spark Core, YARN
• Assignee: Saurabh Chawla
• Reporter: Saurabh Chawla
• Created: 26/Mar/21 14:11
• Updated: 18/Mar/24 13:27
• Resolved: 20/Apr/21 13:57

Description

On Running Spark job with yarn and deployment mode as client, Spark Driver and Spark Application master launch in two separate containers. In various scenarios there is need to see Spark Application master logs to see the resource allocation, Decommissioning status and other information shared between yarn RM and Spark Application master.

Till now the only way to check this by finding the container id of the AM and check the logs either using Yarn utility or Yarn RM Application History server. 

This Jira is for adding the spark AM log link for spark job running in the client mode for yarn. Instead of searching the container id and then find the logs. We can directly check in the Spark UI

Issue Links

• is related to: SPARK-47441 Do not add log link for unmanaged AM in Spark UI
• links to: [Github] Pull Request #31974 (SaurabhChawla100)