[SPARK-20772] Add support for query parameters in redirects on Yarn

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Not A Problem
• Affects Version/s: 2.1.0
• Component/s: Spark Core, YARN
• Assignee: Unassigned
• Reporter: Bjorn Jonsson
• Created: 16/May/17 18:12
• Updated: 17/May/20 18:13
• Resolved: 17/May/17 06:36

Description

Spark uses rewrites of query parameters to paths (http://<fqdn>:4040/jobs/job?id=0 --> http://<fqdn>:4040/jobs/job/?id=0). This works fine in local or standalone mode, but does not work on Yarn (with the org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter filter), where the query parameter is dropped.

The repro steps are:

• Start up the spark-shell and run a job
• Try to access the job details through http://<fqdn>:4040/jobs/job?id=0
• A HTTP ERROR 400 is thrown (requirement failed: missing id parameter)

Going directly through the RM proxy works (does not cause query parameters to be dropped).

Issue Links

• breaks: ZEPPELIN-2221 Spark jobs UI from the paragraph is broken in YARN
• is related to: SPARK-24120 Show `Jobs` page when `jobId` is missing