[ZEPPELIN-2221] Spark jobs UI from the paragraph is broken in YARN

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.7.0, 0.8.0
• Fix Version/s: 0.8.0, 0.9.0

Environment

Zeppelin from 3/1 master snapshot

• Assignee: Jongyoul Lee
• Reporter: Ruslan Dautkhanov
• Created: 06/Mar/17 19:48
• Updated: 24/Dec/20 03:20
• Resolved: 30/May/18 04:05

Description

Discussed on PR-1663 for ZEPPELIN-1692
https://github.com/apache/zeppelin/pull/1663#issuecomment-283477396

When we click on any of Spark jobs UI from the paragraph links,
link is leading to

http://hostname.domain.com:8088/proxy/application_1488384993892_0001/jobs/job/

and that error shows HTTP ERROR 400

Page reads

Problem accessing /jobs/job/.

Reason: requirement failed: Missing id parameter
Powered by Jetty://

A little bit more information - the link on paragprah leads to

http://10.20.32.57:28009/jobs/job?id=123

 Spark Driver UI.
Spark Driver web server then redirects to a link like I sent in my previous post,
and it misses job id in the redirected URL, i.e.

http://host.domain.com:8088/proxy/application_1488384993892_0044/jobs/job/
To your question on consistency - yes it happens in 100% cases, I never saw this new feature works for us.
Is this Spark version dependent or something? We're running CDH Spark 2.

According to https://github.com/apache/spark/pull/5947 URL format is different in YARN and non-YARN modes? Was PR-1663 for ZEPPELIN-1692 tested on both of these modes? Not sure what else might break those links.

Issue Links

• is broken by: YARN-6615 AmIpFilter drops query parameters on redirect
• is broken by: SPARK-20772 Add support for query parameters in redirects on Yarn
• relates to: SPARK-24120 Show `Jobs` page when `jobId` is missing
• links to: GitHub Pull Request #2964
• links to: GitHub Pull Request #3010