Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-20772

Add support for query parameters in redirects on Yarn

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Minor
    • Resolution: Not A Problem
    • Affects Version/s: 2.1.0
    • Fix Version/s: None
    • Component/s: Spark Core, YARN
    • Labels:
      None

      Description

      Spark uses rewrites of query parameters to paths (http://<fqdn>:4040/jobs/job?id=0 --> http://<fqdn>:4040/jobs/job/?id=0). This works fine in local or standalone mode, but does not work on Yarn (with the org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter filter), where the query parameter is dropped.

      The repro steps are:

      • Start up the spark-shell and run a job
      • Try to access the job details through http://<fqdn>:4040/jobs/job?id=0
      • A HTTP ERROR 400 is thrown (requirement failed: missing id parameter)

      Going directly through the RM proxy works (does not cause query parameters to be dropped).

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                bjornjons Bjorn Jonsson
              • Votes:
                1 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: